Hi Nana, hello everybody,

I agree with some of the comments made this far, but here are some additional thoughts.

There are many steps in building a sound ANN (artificial neural network) model to be used with a set of data. In saying this, what I say is that the specifications of the ANN do depend on your data (i.e., your particular problem you want the ANN to solve).

Nana, it is not clear from your description of your ANN what the activation function is (for each unit), and I don't grasp quite well what the cost function is either. I recommend that you use a sigmoid activation function and the cross-entropy cost function, because this combination allows for avoiding a lot of problems (upper- and lower-boundaries problems, and respectively local minima problem).

Then, start with the simplest ANN architecture, that is a 3-layer network. As for the number of hidden layer units, try use the less possible (start with 5, for instance), and allow for more if the network is unable to learn the problem. If the problem requires very complicated nonlinear separations, it may be necessary that you introduce a 2nd hidden layer, but you need to try first with just 1 hidden layer.

Data normalization (if necessary) should be carried out on the full set of your data at once, before you decide which part of your data will be training data and which part of your data will be cross-validation data.

I suggest that you split the full set of your data into 3 sets: training set (about 1/3 of the data), test set (about 1/3 of the data), and cross-validation set (about 1/3 of the data). Split your data set into these 3 sets RANDOMLY. Train your simplest ANN architecture (i.e., 3-layer ANN, with one hidden layer) on the training set until the RMS error drops to 0.1. If the RMS error gets down to 0.1 then you don't need to add hidden units to the hidden layer, otherwise (e.g., if you're stuck with a RMS error of 0.4) you'll need to add hidden units (or, if the problem is really very complicated, even a 2nd hidden layer). However, you don't want the RMS error to go as low as 0.02, as you mention, as this would (possibly) mean over-learning. Over-learning means your trained ANN is very good on the particular set it was trained on, but that comes at the expense of generalization : generalization will be lousy. An RMS of 0.1 is fine at first. Particularly if you get it after about the same number of epochs whatever the training set "chosen". I mean, you should split the full set of your data into the 3 sets mentioned earlier many times, by "chosing" randomly each time the training exemplars that are assigned to each of the 3 sets.

Now, what is the use of the 3 sets. The training set is obviously used to train the network. The test set is used to test generalization during the training phase (i.e., the ANN is not trained on this set, but it is tested on it during the training). If the error curve on the training set separates itself too much from the error curve on the test set, then it means over-specialization (i.e., instead of coming up with a general solution, the ANN picks up a particular solution that works for the training set but not for the test set). This means you have too many hidden layer units. Use an ANN with less hidden layer units and start over. After some fiddling you'll find an ANN architecture that is both able to solve the problem (i.e., is able to learn up to a RMS error criterion -- of, say, 0.05) and to solve it in a way that is not particular to the training set (i.e., that is general enough to encompass the test set). In practice, when you'll get there, the error curve on the training set and the error curve on the test set will be placed (virtually) on top on of the other.

At this point you want to be sure that this is not the result of a particular choice of the exemplars that were assigned to the training and the test set. A first step is to test with the (up to now) never used cross-validation exemplars. If the error on the cross-validation set is about the same as that on the training and test sets, everything is OK (I like at this point having an additional test, which I describe in my following sentence). To make super-sure everything is OK, at this point I take up again the whole set of data and randomly split it into 3 thirds ( training set (about 1/3 of the data), test set (about 1/3 of the data), and cross-validation set (about 1/3 of the data)) and used the ANN architecture that I retained previously just as described above. If you get the same result as above (i.e., once the ANN has learned, the error on the cross-validation set is about the same as that on the training and test sets) then you've nailed it! Otherwise, look for "not-learning" (the error on the training set does not go down, too few hidden layer units) or over-specialization (see above) and fiddle some more.

One more thing: do not chose by yourself the training set, "chose" its exemplars randomly among those of your whole data set.

Also: do not use a too small part of your whole data set as training exemplars (training set); 1/3 is OK, 1/10 is most often than not NOT OK.

Also: ensure your data set is not non-uniformely noisy, that is, ensure there are no data points that are completely aberrant (this has nothing to do with ANNs, you'll have to check it in the real world (I mean make sure what you'll ultimately feed to your ANN is sensible in our world, to begin with)).

Hope this helps.

Cheers,

SCM

The training of the system is mostly dependent on the training input and the types in the training data is presented on neural networks hiden layer you can change the number of neurons and and select the hiden layer neurons with best results.

To give an accurate answer, I need to know the characteristics of the net, the data, the outputs, the way you select the training and the test sets, etc.

From the surface this looks like a bug, a problem with the data or both. The interesting explanation would be the one where the MLP 'found out' that the best predictor for the test data is the mean value.

You can try to train the networks with a different training set; perhaps, speaking from a qualitative point of view, the sample data could be too much correlated each other, so the prediction is the same for each run. Maybe you fall in a local minima, so the result remains trapped in it after few iterations.

Try to check the data, and the way you extract the subsets of training and test set with cross-validation.

Have you tried with standard tests to see if your algorithm is working properly first? It is important always to start with verification steps to have guarantees that the learning algorithms are working properly.

Nana, why do You thing, You got a problem?

Did You consider, that your data yield a good answer with the simple model (one hidden layer), and the second hidden layer is simply superfluous? Do You have some extra evidence, that the second hidden layer simply has no meaninig and should not be

included into the assumed model!

To be sure, You should investigate: how much extra reduction of the error is yielded by the additional layer. This will yield to You the answer, whether the additional layer is necessary in Your model.

Imagine another example: You have two groups of dada, which are linerly separable, which means you need only linear discriminant function to classify the two groups.

Now suppose, you apply for the same data a more complicated model, namely quadradic discriminant function. You will obtain the same result, because the quadratic

discriminative coefficients odfthe quadratic discriminant function are minuscule and exert for your data no additional impact on the quality of the prediction.

So: be happy with Your simplified model, however you should in a convincing way prove, that the accepted model is appropriate for Your data.

I advise strongly to visualise them using some multivariate techniques for visualising

mulivariate data.

Greetings Anna

Thanks for all these responses,

the data that I use is downloaded from this site:http://datamarket.com/data/list ?q=provider:tsdl%20cat_id:20814

so I've normalized my input data in [0.1],

for the first time I've used MLP with one hidden layer, off course with several configuration, it gives me an error around 2E-2, but at the test the prediction is almost steady, so I've thought that may be if I use an MLP with more than one hidden layer, this may be will improve the prediction, but unfortunately it is note the case.

for the cross validation, because the data set size = 19213, so I've used 5 folds cross validation, the error during the training for each set is about 2.XXXE-4

I've used one software with this data, and it gives me a good result of prediction, so I do not think that there is a problem with the data.

finally, for the hidden layer(s) some times I use tangh and sigmoid, sigmoid and sigmoid, tangh and tangh...( of course for the multi hiddenlayer)

thanks and best regards.

Nana, maybe your sigmoids are saturated which would explain that the output value remains constant ? Are your test data normalized on the same way than the training data ?

Nana,

Make sure that you use a good randomization function with different seeds. Make sure that the cross validation function is really working and also check that the initial weights are different. By printing the initial weights and how the data is set up before running the network will provide you some idea that the implemented functions of initialization are working correctly. I would also suggest printing the first 10 iterations of the weights and track their changes by doing a manual computation to see if there are bugs in the program (choose a small problem:-).

Hope you find the cause of your problem.

Regards

Yes my test data is normalized, with the same way as the training data

Is it a classification or regression task? If regression,what is the range of the output data?

the range of output data is [0,1], since I use the sigmoid function in the output layer

just one neuron in the output layer

Don't take offense, but are you certain you are feeding the test data to the _trained_ version of the network - it might require a software switch change? MLP's are pretty straightforward, so a software bug is a possible explanation.

A common mistake is do the normalization of the entire dataset at one time. If this happened to you, you should normalize the 3 subdataset (training, crossvalidation, test) independently to avoid any data "pollution".

If there aren't any problem with the implementation, initialization, dataset or normalization, the most probably cause is that your hidden layers are oversized. I would try with smaller hidden layers and increase them, until error becomes stable.

Ok, I train my network several times with randomly chosen weights [0,1]

and for time I save the optimal weights and configuration (nbr of hidden layer),

at the and of training, I choose the configuration with the smallest error MSE, and for a test I load my network with this optimal configuration (weights and nbr of hidden layer), then I present my test set to the chosen network.

for the normalization Ive tried with two techniques , the min max technique and the mean sigma technique.

Regards.

What is the order of your error in the test set?

The test error = 0.0215439,

I believe that the problem is in the data, can you make available the dataset so that everyone can understand what kind of data are you using?

Are learning data representative, are they very noisy?

It seems to me that learning was not successful due to the demanding training data.

Is it possible to obtain more training data?

Is it possible to remove noise from the data?

The test error is 100 times worst than training error, so your model probably was overfitting the training data. In other words, your model is so complex that it's fitting (or memorizing) the error in the training data.

The solutions, Bojan Ploj mentions some of them, can be:

- Reduce the model complexity (smaller hidden layers). You should see more training error, but less cross-validation/test error. Keep the reducing the model until cross-validation error stop to lowering.

- Use some kind regularization in the cost function. It will penalize the model complexity and it helps to avoid overfitting.

- Obtain more training data. You can use bootstrap if more data is not available. In some cases you can do some transformations.

- "Remove noise"

@Graciano Pousa

I agree with your suggestion that the models may both be overfitting the training data.

I voted you up on that post. :-)

However, I am not sure that each data set (training, cross-validation, test) should be normalised separately. My understanding is that the training set should be normalised and the same normalising transform should be applied to subsequent data submitted to the network. So the cross validation and test sets will be subject to the same translation and scaling applied to normalise the training set.

About the normalization, by separately, I mean that If you are using test data to normalising training data, your model is not independent of the test data.

Hi Nana, hello everybody,

I agree with some of the comments made this far, but here are some additional thoughts.

There are many steps in building a sound ANN (artificial neural network) model to be used with a set of data. In saying this, what I say is that the specifications of the ANN do depend on your data (i.e., your particular problem you want the ANN to solve).

Nana, it is not clear from your description of your ANN what the activation function is (for each unit), and I don't grasp quite well what the cost function is either. I recommend that you use a sigmoid activation function and the cross-entropy cost function, because this combination allows for avoiding a lot of problems (upper- and lower-boundaries problems, and respectively local minima problem).

Then, start with the simplest ANN architecture, that is a 3-layer network. As for the number of hidden layer units, try use the less possible (start with 5, for instance), and allow for more if the network is unable to learn the problem. If the problem requires very complicated nonlinear separations, it may be necessary that you introduce a 2nd hidden layer, but you need to try first with just 1 hidden layer.

Data normalization (if necessary) should be carried out on the full set of your data at once, before you decide which part of your data will be training data and which part of your data will be cross-validation data.

I suggest that you split the full set of your data into 3 sets: training set (about 1/3 of the data), test set (about 1/3 of the data), and cross-validation set (about 1/3 of the data). Split your data set into these 3 sets RANDOMLY. Train your simplest ANN architecture (i.e., 3-layer ANN, with one hidden layer) on the training set until the RMS error drops to 0.1. If the RMS error gets down to 0.1 then you don't need to add hidden units to the hidden layer, otherwise (e.g., if you're stuck with a RMS error of 0.4) you'll need to add hidden units (or, if the problem is really very complicated, even a 2nd hidden layer). However, you don't want the RMS error to go as low as 0.02, as you mention, as this would (possibly) mean over-learning. Over-learning means your trained ANN is very good on the particular set it was trained on, but that comes at the expense of generalization : generalization will be lousy. An RMS of 0.1 is fine at first. Particularly if you get it after about the same number of epochs whatever the training set "chosen". I mean, you should split the full set of your data into the 3 sets mentioned earlier many times, by "chosing" randomly each time the training exemplars that are assigned to each of the 3 sets.

Now, what is the use of the 3 sets. The training set is obviously used to train the network. The test set is used to test generalization during the training phase (i.e., the ANN is not trained on this set, but it is tested on it during the training). If the error curve on the training set separates itself too much from the error curve on the test set, then it means over-specialization (i.e., instead of coming up with a general solution, the ANN picks up a particular solution that works for the training set but not for the test set). This means you have too many hidden layer units. Use an ANN with less hidden layer units and start over. After some fiddling you'll find an ANN architecture that is both able to solve the problem (i.e., is able to learn up to a RMS error criterion -- of, say, 0.05) and to solve it in a way that is not particular to the training set (i.e., that is general enough to encompass the test set). In practice, when you'll get there, the error curve on the training set and the error curve on the test set will be placed (virtually) on top on of the other.

At this point you want to be sure that this is not the result of a particular choice of the exemplars that were assigned to the training and the test set. A first step is to test with the (up to now) never used cross-validation exemplars. If the error on the cross-validation set is about the same as that on the training and test sets, everything is OK (I like at this point having an additional test, which I describe in my following sentence). To make super-sure everything is OK, at this point I take up again the whole set of data and randomly split it into 3 thirds ( training set (about 1/3 of the data), test set (about 1/3 of the data), and cross-validation set (about 1/3 of the data)) and used the ANN architecture that I retained previously just as described above. If you get the same result as above (i.e., once the ANN has learned, the error on the cross-validation set is about the same as that on the training and test sets) then you've nailed it! Otherwise, look for "not-learning" (the error on the training set does not go down, too few hidden layer units) or over-specialization (see above) and fiddle some more.

One more thing: do not chose by yourself the training set, "chose" its exemplars randomly among those of your whole data set.

Also: do not use a too small part of your whole data set as training exemplars (training set); 1/3 is OK, 1/10 is most often than not NOT OK.

Also: ensure your data set is not non-uniformely noisy, that is, ensure there are no data points that are completely aberrant (this has nothing to do with ANNs, you'll have to check it in the real world (I mean make sure what you'll ultimately feed to your ANN is sensible in our world, to begin with)).

Hope this helps.

Cheers,

SCM

Thank you so much for all this responses,

may be the problem is with my training method:

because, I use directly the cross validation with 5 folds, so I divide my data (after normalization) in 2 sets, 75% for the cross validation (which mean I divide this set to 5 other sets each one contains a training set and validation set) and 25% for the test set.

First I train my networks(many topologies with different rates and momentums values) with the the first set of the cross validations set, then I test it with the validation set of this first fold, and I do the same thing with the other sets,

of course for each training I've put a threshold of 2E-4 (MSE), then I save the weights of the optimal network.

then I load again the network with the saved weights and I present the test data set to the network.

Are you familiar with plots like the one I've linked?

You may have a problem as simple as training your networks too long. I would try saving the weights every epoch during training (or every 10 epochs, or 100, or whatever, depending on how many total iterations you're using for training) and evaluating each of them using your test set. It may be that you've over-trained your networks and would be better off with fewer epochs.

http://en.wikipedia.org/wiki/File:Overfitting_svg.svg

@Nana: Firstly, cross-validation and test sets are not to be used to train the network; I spelled it out quite clearly in my previous post. From what I understand from your last post it seems to me that you are using too small a training set. This means your ANN picks up a solution that is particular to the (rather small) particular training set you use, and by no means general enough. That is, you get overlearning on the training set and lousy generalization. Take your time to read my previous (and quite long) post and let me know if you have any questions. If you follow the recommendations I made in that post, you should not run into the problems you describe.

@Jared: You are perfectly right. Thanks for supplying an illustration of the overlearning problem. However, the way the figure looks, it is not quite an illustration of the overlearning problem as much as an illustration of what happens when your learning set is not representative of the whole training set (or problem, if you like), that is, a learning set too little or not randomly "chosen" from the whole set. Were it an illustration of the overlearning problem, the training curve for the test items (red) would have been pretty close (or, virtually on top of) the training curve for the training items (blue) for the initial part (closer than what they are in this illustration), then the red curve would have stopped dropping while the blue would continue to drop, thus the 2 curves would have separated one from another (but in a specific way, slightly different from the one depicted in the figure you provide).

Nana, HTH.

Cheers,

SCM

Thanks for all, for all responses, I've followed your advises, and I really appreciate your collaborations.

My problem is almost resolved, now the prediction changes with the same way as the target values, it is not perfect yet but it is much more better than before.

Still I hope you solution gives over-fitting results. Do statistical analysis on data set & then decide the algorithm that suits the data.

Overfitting case..................and normalization of data must be done

please guide how rminer/nnet set initial weight in MLP. Does this initial weight matter in training if we use default?

Manojit Chattopadhyay :

hi

you can find your answer at fausett's book page=296

=>> choice of initial weights and biases

Hello

Maybe you have not normalized the data. So the sigmoids of the hidden layer are saturated and the output of the hidden neurons are constant.

You may require normalization of input data. Also if your input data has more inter-class similarity, that data requires pre-processing. 

If your target values are not binary values, try to train for binary output values.

There is also the possibility of falling into a local trap.

Try with different parameters values in your learning algorithm, i.e if you used PSO for your NNs, increase its inertia parameter values, to help for a jump from the trap.

Dear Nana,

Musca is right in his view.

In addition with all the answers, you can check what is happening inside on your network by simply visualize your network and log via tensorboard.

I think from there you can point out.

TIA

The trained network is for certain weights, for further optimization use solver, you'll see dramatic changes in the values also R square values come down further indicating the network trained to predict well with error rate minimized to the best possible extent..

What is happening if we take same data for input and target output in Artificial Neural Networks.

According to me, the output changes in function of the presented innput.

Agreed, I too have used solver and found the weight change bring about the optimization.

All the best for your project dear

You can also check if the number of hidden neurons is appropiate

i think that Musca is right