This may be a stupid question, but I was just wondering why you guys use a convex combination of the two regularisers? This isn't necessary to maintain convexity. Am I correct in saying that any linear combination of the regularizers will still form a convex problem?
I can see that the alpha effectively weighs in group sparsity or more general sparsity, but what if we want to look at a case where we have high group sparsity and high general sparsity? Did you consider treating the regularizers with different lambdas?
I have seen the convex combination (1-alpha) R1 +alpha*R2 used for regularization in other papers also, so perhaps there is something obvious I am missing.
Nice paper though!
Article A Sparse-Group Lasso
Any non-negative linear combination of two convex functions will result in a convex function, yes. The reason why the combinations are normalized I think has to do with the fact that there is little point in moving outside of the set defined by non-negative weights summing exactly to 1. (Once you have picked ANY weights you can divide the resulting function by the sum of these weights, resulting in a sum of weights of 1, and this division does not change the set of optimal solutions.) If you however allow one of the convex functions to have a negative weighting factor, you risk severely having the summed-up function being non-convex. And we do NOT want to deal with those.
To write an abritrary nonnegative combination a_1 R_1 + a_2 R_2 as
\lambda (\alpha R_1 + (1-\alpha) R_2) happens to give a nice interpretation to the coefficients: \alpha as the 'mixing parameter' of the two regularisers and \lambda as the overall regularisation. Of course, it doesn't really matter how you write it as long as there are two parameters...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics