I'm using the feed forward neural network with 100 inputs and 10 units in hidden layer and one output neural net. I train the network several times using the same input training data and the same network architecture/settings with random initialize but I understand that there will be differences in the weightings produced within the NN each time and that no two neural networks will be identical, but what can I try to produce networks that are more consistent across each train, given the identical data?
In Neural Network, Initial weight is very important. so when training (with matlab) save best initial weights that produce best and fast result.
You should use bagging. It is meant to transform unstable learning algorithms into stable ones. You will end up with an ensemble of ~10 different neural netwok, where the average of the answers will be stable. Of course your ensemble would be different from one training to another. But your set of answers on the test will be more accurate and more stable.
Apart from bagging you might want to consider using ensembles over several training initialisations, which are somewhat easier to implement. Both bagging and training ensembles should provide relative stability of the results and improvements in the performance over a single neural network.
For a reference have a look at :
https://www.researchgate.net/publication/259165043_Neural_Network_Ensemble_Operators_for_Time_Series_Forecasting?ev=prf_pub
Article Neural Network Ensemble Operators for Time Series Forecasting
Bagging is simple to use.
You can refer to wikipedia for a description of the algorithm.
http://en.wikipedia.org/wiki/Bootstrap_aggregating
(sorry for this badly formatted pseudocode, there is not much I can do with this text editor. The leading periods serves as indentation.)
# at learning time
predictor_list = [ ]
repeat N times: # N ~ 10
. . . train_set_ = sample_with_replacement(train_set)
. . . predictor = MLP.fit( train_set_ )
. . . predictor_list.append( predictor )
# at prediction time
y_list = []
for predictor in predictor_list:
. . . y_list.append( predictor.predict( test_set.x ) )
Finally, you can see y_list as samples from the probability distribution over the answer. To select the optimal answer, you should average your answer according to the optimal decision theory. In simpler words : use the most frequently predicted class in the case of a classification problem (zero-one loss) or use the average answer in the case of regression (square difference loss).
If you using Mathlab...same problem to me. when it's first time train up training functions occupied memory and this is not erased until closed the window of Mathlab. For retraining memory overlapped with previous memory......i think for this reason it shows different result for different times.
- Badges
- Science topic