Support Vector Machines (SVM), Logistic Regression (LR) and some other classification methods can guarantee, that obtained solution really will be global minimum, because they use convex goal functions. Neural Networks algorithms suggest to use different goal functions (Cross-Entropy, MAE, MSE, etc.), but it seems me, that all types of goal functions are not convex. So, question - how Neural Networks can guarantee, that obtained solution really will be global minimum? Thanks beforehand for your answers. Regards, Sergey.
I think this depends on various factors; the nature of data, the network topology, and the learning rate of the network and error percentage, as I understood the question.
Dear Mohammed, I fully agree with you, that "it depends on various factors". So, sometimes it will be good convergence for global minimum, and sometimes - bad convergence. But we don't use on NN '"Global optimization algorithms" (such as Genetic, Simulated Anneling, Tabu , etc.), so how NN can provide really optimal solution?
Dear Sergey,
There are many options need to be optimized to ensure that we are using the most appropriate form of a specific model. For example, MLPNN model, and about the convergence you mentioned in comments, this models can be trained by various training algorithms such as LM (Levenberg-Marquardt) or GD (Gradient descent) and etc. It has been ascertained that the LM is usually preferred due to its better performance. Other parameters can be evaluate with several trial and error procedures.
Also, if you re-optimize the proposed model by means of optimization algorithms, such as GA or PSO and ..., I think it can be guaranteed that you are using the most reliable form.
Dear Mohammad, I fully agree with you, that using of Global Optimization Methods (such as GA, CE, SimAnnel, etc.) can guarantee obtaining of global/real minimum, but is it really possible to use them during NN tuning? They don't work fast, rather slow -:)
That's an old question indeed. I conjecture that the electrical power wasted in trying to find global minima by training neural network with genetic algorithms, simulated annealing, and similar heuristics, during the past thirty years, is one of the factors of global warming...
Bad news : no one can ascertain, in humanly acceptable computer time, that the global minimum of the training error has been reached by the optimization algorithm at the end of training.
Good news : there is definitely no point in trying to find the global minimum of the training error. The purpose of machine learning (neural nets or other methods) is not to obtain a model that fits perfectly the training data, but to find the model that generalizes best, given the training data. If you train a lot of neural networks from the same data, with different parameter initializations, and estimate their generalization errors e.g. by hold-out or cross-validation), you will get a large number of neural nets, and you will find that the neural network that has the smallest training error is not the network that generalizes best. Actually, a popular regularization method in neural network training is known as "early stopping", which consists in monitoring the generalization error on a validation data set during training, and stopping the optimization when the estimated generalization error increases, before reaching a (local or global) minimum. In my opinion, a more reasonable procedure consists is training as many neural networks as possible given your allotted computation time (with different parameter initialization and different complexities), in estimating their generalization error (e.g. by cross-validation), and in selecting the model (or an ensemble of models) exhibiting the smallest generalization error. You can find a systematic methodology of neural network model design in the textbook "Neural networks, methodology and applications" (Springer, 2005).
The only guarantee that you found the global minimum is that the function value of your objective function (cross-entropy, L2, or whatever you use) is zero. Since the misfit functions are non-negative, you know that if your misfit is zero, you found the global minimum. However, keep in mind that the global minimum might not be unique, so there might be a large number of different sets of parameters, that give you a zero misfit. Also, as pointed out by the previous poster, a zero function value does not say anything about how well your network will perform on the test data set.
Early stopping is considered a faux-pas by optimizers though and to avoid overfitting or to address non-uniqueness, you should use regularization instead.
- Badges
- Science topic