in back propagation neural network training, using gradient decent, the derivative of weights with respect to error are calculated by back propagation algorithm.is this calculated value the exact gradient or it is the approximated gradient of error with respect to weights for each input?
if you approximate the expected value of the error by the instantaneous value then the calculation is an approximate...
There is no exact here , because back propagation is first order derivative of the system and we know machines can only give us approximated values for a differential form.
Having said that - gradient in itself has to be converging and for convergence there are times when the feedback values are approximated (not visible outside the NN)
why it is an approximated value? for example if I have analytical function of error (that is the function of biases, weights and inputs) and differentiate the error with respect to a one of the layer weights (then substitute the values of weights and inputs in order to get the values of gradient ) this obtained gradient is not exactly the same as the one obtained form back propagation algorithm?
Well Ali , back- propagation or gradient descent is meant for convergence !! Why convergence - because it is a network or flow of certain features and not the "raw" data per se .
Now , with features how do we ensure proper learning is done -trial and error . This is why weights can be fine tuned / bias can be changes so the gradient descends towards a stable value called convergence point .
Basically , it all starts with estimation so can it be unbiased or should be biased ?
What are the advantages of biased estimation vs unbiased ?
Because it is back-propagation there is feedback that enable correction or minimization of error as features travel thru the network (unlike feed forward ) .
It is definitely approximate. It is rate of change of error with respect to change in weight. Here, error is the difference between actual value and expected value of the output. Weight has its origin in some random value and it is not tailor made. It gets stabilized repeatedly over iterations.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training