I believe the problem is that for convex optimization we need a mechanistic objective function, concave or convex and a convex set as the constraint set, in most deep learning situations the mathematical form of the error surface is not known apriori, hence one resorts to numerical optimization methods like conjugate gradient method or nature inspired optimization methods like Genetic algorithms in such situation.

In deep learning the surface is very non-convex, and convergence is not guaranteed, still less convergence to a useful results. However, regularization improves the shape of the loss function and increases the chances of convergence.

In deep learning, there are several optimization problems. One for each layer weights and biases. What I’m curious about is can you solve a bunch of convex optimization problems such that your non-convex objective function always converges to a solution. I havent worked this out mathematically, but i was just curious if anyone has.

I guess what im asking if if your objective function gradient is non-convex w.r.t the input, is it possible to break it down such that the objective function w.r.t. the weights and biases are convex?