I am trying to train CIFAR10 dataset on CIFAR10 architecture on MATCONVNET without gpu but my objective function is not decreasing down to 1,665. What may be parameters to tune while considering theses graph?
These option where considered (Momentum=0.9; BbatchSize=256; NumEpochs=45; Continue=true; …) and the rest were default.
The question is very important. Go back to a prominent topic of the 1970-90s: Simulated Annealing.
Let me explain with a physical analogy.
In statistical physics, particles have average speed v, where the average temperature is such that v =sqrt(3kT/m).
You increase the temperature to get particles moving, in particular this causes the system to get out of local minima traps. You cool down the system for things to settle, and get stability.
In simulation, add interaction forces between your system parts as for statistical physics between particles at mutual distance r, mass m. Physics has studied stable systems at play in solids. You know Newtonian and Colombian forces, try Van der Wals in r^(-7) which are excellent for stabilising in simulations, and try a Yukawa potential to screen effects beyond a certain close distance.
I have been quite happy with these when I had a problem of the kind you described. Mine was to spread points on a sphere, getting rid of local minima...
Hope it works for you
Dear Djimeli TSAJIO Alain Bernard,
There are a lot of reasons why this might be happening.
- But since you are using a built-in model within a library of reference, I narrow down the field of possibilities based on the parameters I see.
- There is a strong interplay among batch size, learning rate, learning speed, nber of epochs and learning stability
- When I see the figure you attached, the learning process seems not to converge to 1.66, it is just to slow and get stopped after 40 epochs.
- Your batch size is too large, even the number of epochs is too small, and your learning rate is probably to small
- Reduce the batch size to 32 or smaller, reduce the learning rate too around 0.01-0.001 and increase the number of epochs hundreds
The use of global optimisation is the best here, I think, but it takes a while to understand it, and master it - so it is a matter of whether you have the stamina to teach yourself this technique.
The question is very important. Go back to a prominent topic of the 1970-90s: Simulated Annealing.
Let me explain with a physical analogy.
In statistical physics, particles have average speed v, where the average temperature is such that v =sqrt(3kT/m).
You increase the temperature to get particles moving, in particular this causes the system to get out of local minima traps. You cool down the system for things to settle, and get stability.
In simulation, add interaction forces between your system parts as for statistical physics between particles at mutual distance r, mass m. Physics has studied stable systems at play in solids. You know Newtonian and Colombian forces, try Van der Wals in r^(-7) which are excellent for stabilising in simulations, and try a Yukawa potential to screen effects beyond a certain close distance.
I have been quite happy with these when I had a problem of the kind you described. Mine was to spread points on a sphere, getting rid of local minima...
Hope it works for you
A tuning parameter (λ), sometimes called a penalty parameter, controls the strength of the penalty term in ridge regression and lasso regression. It is basically the amount of shrinkage, where data values are shrunk towards a central point, like the mean.
In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned.
Tuning Machine Learning Models. Tuning is the process of maximizing a model's performance without overfitting or creating too high of a variance. ... Hyperparameters differ from other model parameters in that they are not learned by the model automatically through training methods.
https://www.mathworks.com/help/control/examples/tuning-control-system-with-multiple-valued-plant-parameters-using-control-system-tuner.html
Thank you for your kind quick response. I take them into consideration and will communicate the results to you soon.
I am interested in the approach proposed by Professor Michael Patriksson and I look forward to receiving a link for the understanding of the global optimization approach.
Dear Djimeli TSAJIO Alain Bernard,
There is no such practical global optimization approach. The way neural networks are trained is the fruit of decades of research. It is not because we do not know about global optimization: It is obvious that you are looking for a global extremum as anyone else.
Very interesting question. Following it. I agree with Franklin Kenghagho Kenfack
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs