Hello, Prashant

Even if the most likely reason is a programming bug, Renan's tip about the ground nodes is noteworthy. You may try them in the output layer as well.

Also, it could be a representational capacity problem, as Marco says. In this regard, I would recommend you to make a Principal Component Analysis (PCA) of your training data. In case your goal were to make a classification of the training examples (which is likely given your network's architecture), the PCA may give you a hint about the number of different dimensions that must be discriminated by hidden units. In case your hidden layer had too few units, the distributed representations of input vectors they learn might not be different enough to reduce the network's error.

In any case, you could provide the error VS epochs graph as it may convey some information.

Can u give more details in the sense, does it happen only after 30 epochs or it may happen before too in some cases..

Have you tried to flush /reset the system as it may happen that some information might be cached. However, if this is not the problem then please provide more details about your problem

What about the trainning data? What kind of architecture are you using? Is the error high? We would need further information... but perhaps it is related with a overfitting problem (simple data / complex architecture) so the neural network is really memorized the input data. As Herman said, it should be related also with a fixed initialization of the weight values so the trainning is a deterministic process.

I think you need to provide some more information i.e. the ANN software you're using (is it the matlab toolbox?) and how you're training your net. Also, what training algorithm you're using (LM, GDM, etc.)

I have developed my own network in java without any library.

I have tested my network for Logical Gates it works fine(I have not used momentum for this). I am using simple backpropagation(gradient descent).

My data set contains almost 1200 samples and i have normalized inputs & outputs.

Weight update method is online updation(after each sample) activation function is sigmoid for both input and output.

Architecture is 5 inputs ,1 hidden layer and 1 output.

Regardless of number of hidden neurons or learning rate or momentum network produces same MSE after 30th epoch. MSE varies with each change but remains same after 30th epoch and value of MSE is quite high in all the cases.

I am unable to find problem.

Now i tried network with two hidden layers and it is working fine(not perfect but at least error reduces with each epoch and it does stops after 30th epoch).

But still working on my previous network to find problem.

First try with a standard package for neural network (in R, Matlab...) to understand if the problem is coming from your data or your java code.

Try to change the parameters of the activation function. If use a sigmoid function, then is the changing of the rise a good part, or select a suitable interval for using a arcthan()/tan() function. Sometime is helpful to implement a own activation function and not use the standard functions from a library.

Prashant,

I would suggest that you plot the error rate for every output and see if there is anything unusual. I would then match the error to the input output pair at each point to see if it gives what it should(take the first 10-20 input output pairs and that should give you a good idea of how the NN is doing).

a simple and useful trick : check your code on data generated from the neural network architecture you are using.

for instance, if you use a MLP with 3 input, 5 hidden and one output units, generate a set of training patterns from a (3/5/1) MLP ("master "MLP) with a fixed set of weights (randomly chosen, for instance ... or as you like), then train a (3/5/1) MLP ("student" MLP) with your code on these patterns. This shoud lead you to an asymptotically small error rate (although the weights themselves of the "student" MLP could be different from the weights of the "master" MLP). If the error rate still saturates at some point (at a non negligible value from a machine point of view) ... check your code !

..

err, 30 epoch for 1200 n data? Quite a fix you got there. No offense :P :)

Assuming no error in your code (which IMHO, it is likely), try several fix here:

- try to provide additional ground node (node that always 1) in input and hidden layer.

- Make sure every node is a double variable, and already instantiated. My student made the mistake by storing them as integer. Yikes. :)

- Make sure the report is in double. This also common error. :P

- Make sure input and target / testing data set is different. Test the system on the testing data set on every epoch. Count system accuracy to target, not training dataset. Normally student made this mistakes.

- start from 2 hidden node, and when hitting a minima, add a hidden node and start over. More on the concept from Cascade Correlation Neural Network, which IMHO also true for Back Propagation. Normally in my lab the ideal node is around 2 * input * output + 1, e.g., 5-21-2 or 5-11-1, but I am not sure this rule of thumb is proven.

Actually i have tried my network for simple examples like Logical gate it works fine, i also have tried different network structures, now i think the problem is with my input data or normalization technique i have used for my input data.

Now trying some different normalization technique can you suggest some good techniques? I have used min-max normalization technique previously.

err, is there any normalization technique other than "thermo converter" type (may be this is what you call min max?)? Unless it is a code error, it is very unlikely someone go wrong with the normalization.

I just curious, can you show me the output / console print here?

may be something like table below.

epoch accuracy

1 0.1

2 0.09

3 0.0892

...

The normalization depends on the data you process.

For continuous values, you can standardize your variables by subtracting the mean value and dividing the substraction by the standard deviation. Its usually better than Min Max. Indeed if you have some extreme values far from the point cloud, you might flatten the differences.

For symbolic values you can use a complete disjunctive coding.

But, I repeat again, its depends on the data. For image processing, the normalization will be different. It is a way to introduce prior knowledge.

I have stock price(i guess continuous data like time series) as my input data can you suggest any technique?

I never worked on stock price data.

However, in a past life I worked on network data, which are often time series data.

You should take into account that your data are time series, first in building aggregate such as mean on a sliding windows to smooth your serie. Then, you have to study seasonalities and trends to substract them from your time serie, else your problem is not stationary. Then you can use a standardization such as x'=(x- mean (x) ) / standard deviation (x).

"Now i tried network with two hidden layers and it is working fine"

Couldn't it be that one layer is simply not enough? That after 30 epochs with only one layer the ANN gets to the best accuracy attainable and stops there?

Hello, Prashant

Even if the most likely reason is a programming bug, Renan's tip about the ground nodes is noteworthy. You may try them in the output layer as well.

Also, it could be a representational capacity problem, as Marco says. In this regard, I would recommend you to make a Principal Component Analysis (PCA) of your training data. In case your goal were to make a classification of the training examples (which is likely given your network's architecture), the PCA may give you a hint about the number of different dimensions that must be discriminated by hidden units. In case your hidden layer had too few units, the distributed representations of input vectors they learn might not be different enough to reduce the network's error.

In any case, you could provide the error VS epochs graph as it may convey some information.

Sounds much as if the "hidden" units do not have nonlinear transfer functions. If the code is otherwise correct, one layer is theoretically enough and continuing training will lead---if there is not a ridiculous small number of hidden units in one layer---to different errors. They may not be as low as desired, but differ.

One layer is enough for representing any Borel-measurable function, as shown by Hornik/Stinchcombe/White and Funahashi in 1990 or thereabout. (Note: That does not mean that more layers doesn't add more!)

It is also not clear how the (time series) data is input. If the time series data is input in a 5-value time window, that sounds like very little. 1000 data is very little. Etc.

Finally, don't do stock market prediction with NNs. It's such a bore. Better make money with selling stock market prediction software to others.