I am using a 2 layers (one hidden layer ) Neural Networks based classifier to run a classification for my data (images) and use back propagation algorithm.
the networks works well and can classify with more than 90% of accuracy . But the shape of weights in hidden layer are different by running every time. I am using Matlab imagesc function to visualize the weights.
Running every time what? The weights don't have be so or so, they only need to be adjusted to minimize (in general, locally) the residual. Nothing else.
Let me to explain more.I have a data set with 40000 sample. I train the system with this data with initially random weights . at the the end ( after convergence ) I have a trained weights ( let me name them W1 ).
then try to start from beginning . and at this time I have the final weights W2.
W1 and W2 are not the same. They have different shapes?
why?
The answer is evident -- it is in your question. You start at random initial conditions. As Amir also pointed out, the result does not have to be the same if the residual is a non-convex function of weights (and it really is in many cases).
It is usual. every time you initiate weights randomly. therefore, the final values are not same. But the accuracy and performance approximately is the same.
For any interestingly difficult problem there will be many local optima (good, but not globally optimal) solutions that back propagation can find. I also suspect, but it's not proven, that for most simple problems there are many symmetric, equivalent solutions that back propagation can find. Thus, if there's any random step in your optimization algorithm it's going to find different solutions on different runs/trials.
There are basically only two ways to solve this. As Amir suggests you can seed the random number generator with the same value each time. Then you'll get the exact same set of random numbers in each run and they should be identical.
Alternatively, you can remove all randomness from the algorithm (making it deterministic): start with the same initial weights (not random ones), always test the training cases in the same order, etc.
However, I'm not sure why you would want to do that. If there are multiple optimum I would think you want different results on different runs/trials, so that you have a better chance of finding the global optimum. Rerunning the exact same deterministic algorithm over and over won't accomplish anything.
I think the cost function of Multi Layer Perceptron (MLP) is not convex. hen train the nn with train set and evaluate network generalization on test set. Each initial weights may lead you to different learned model. Then select the better one by considering the generalization power. If your data is imbalanced I suggest you to use different performance measure instead of accuracy. So it is possible the cost function has several local optima. So the NN is very sensitive to initial weights of connections.
An ANN is trained with randomised intial weights, that's why different starting conditions converge on different potential solutions. Take a look at this for the influence of the final layer on previous layers:
McLean, D., Z. Bandar, and J. D. O’Shea. "The evolution of a feedforward neural network trained under backpropagation." Artificial Neural Nets and Genetic Algorithms. Springer Vienna, 1998.
The journal website lets you see the first two pages for nothing, unfortunately there isn't a free online download at the momnent.
Try to use a "Border Pairs Method" (BPM ) instead of "Backpropagation" and you will get better results and equal weights every time. BPM method is also constructive and good for noise elimination.
Article Border Pairs Method—constructive MLP learning classification algorithm
Chapter Advances in Machine Learning Research
I completely agree with the answer given by James Dominic O'Shea that weights are initialized randomly each time you start learning phase in ANN. These randomly assigned weights lead to different convergence states and that is why we get different classification/prediction accuracy every time.
Yes , In normal way , These randomly assigned weights lead to different convergence states. How can it converge to one state by doing some constraints?
I have also the same question. I am running the k fold cross validation to find the optimum number of neurons in neural network, for that i need to decide some measure to quantify when my neural network is trained. My criterion is some threshold for training errror and validation error. For that i "train" the model 10 times and take the model which gives me minimum testing error. However i am not sure why i should take only 10 such iterations. Any help will be appreciated
- Badges
- Science topic