I have a Recurrent Neural Network with architecture 5(inputs)-4(hidden)-1(output) with output connected to context neuron only (only one feedback from output neuron). I am using Back Propagation through time.
I have understood forward pass and calculated it. Now in backward pass i have calculate error and delta of error but now how to update weights?
Whether BackPropagation of is applied to all layers or just for context layer to output or to whole network?
Actually I have some problem in how to implement it in java? I have created array of 5 network (for 5 unfold) for each network calculated i submit its output to context neuron of next network and do forward pass for each network.
For Backward pass I am calculating error of 5th network and delta of network, for remaining networks I add calulate error as
err=error(desired -output for respective input)+weight from context neuron to output of next network* delta of error of next layer,
but now whether I have to apply full BackPropagation to each network or just update wt of context layer to output and weights of hidden to output of each network?
I am stuck at this point please help me.
And please suggest any sample dataset like time series on which I can test my network.
My aim is time series prediction with recurrent neural network.
Lets use BPN (Back Propagation Network) as abbreviation.
For starter, which recurrent network you talking about? Jordan Network or Hopfield Network? My answer below assume the first one, which normally used to provide series of answers. Consult to this book, http://books.google.co.id/books/about/Artificial_intelligence.html?id=eH9QAAAAMAAJ&redir_esc=y
IMHO,
1. In concept, New Weight is updated in back propagation as Old Weight * Difference from Old Output to Target Output.
2. I have created array of 5 network (for 5 unfold)" You mean, You have 5 BPN lets say, A, B, C, D, (, and E?), and You use the first 5 as input to E? And You got proper training set for each BPN? (This guess assuming we talk about 5-4-1 network, which 1 input and hidden node always filled with 1 to provide ground)
If this true, then
2.1. If E training set can be calculated from input, or you have training samples for every possible input combi for E, then you can train all BPNs in 1 go.
2.2. If E has her own training set that the relation is incomputable, then no, all BPN should be trained separately.
But, can you elaborate why the network should be 5-4-1 in the first place? Normally, in my lab / study, we decide on the number (n) of nodes in layer in following way:
- n - Input nodes equal to n - input parameter.
- n - Output nodes equal to n - output parameter.
- n - hidden layer equal to power estimation of relationship between input and output, e.g., 1 hidden layer for linear, 2 for ^ 2. Still unproven though
- n - hidden nodes is start at 2 (or 1), and subsequently added when hitting a minima that bigger that target accuracy. Consult on Cascade Correlation Neural Network Concept, which part of it can be used for BPN.
Thanks Renan, Yes you got it correct for 5 arrays means 5 unfolds but for network architecture 5-4-1 i mean it is just for example its not fixed thought inputs node and output nodes are fixed as 5 & 4 respectively, but hidden nodes are fixed.
I am also thinking of taking a network with 2 hidden layer so need your suggestion on it whether i should go for it or single hidden layer is fine?
Also i need some more information about BPTT, because in BPTT we use batch update and update weights only after computing error of network at each unfold in my case 5.
But can i update the weights of each network with respective error at each unfold like in online learning or pattern learning, will it work?
"I am also thinking of taking a network with 2 hidden layer so need your suggestion on it whether i should go for it or single hidden layer is fine?"
I used to believe that number of hidden layer equal to the power of relationship between input and output, lets say, single layer if the relationship is linear, 2 hidden layer for quadratic. Need to proof that. :)
for subsequent question, i don't have an experience with BPTT yet, but, assuming that every ANN you use is independent to each other other than input and output, yes, you can.
Refer Improving forecast accuracy of wind speed using wavelet transform and neural networks by Ramesh Babu. N, Arulmozhivarman. P available in researchgate link for more details.
- Badges
- Science topic