Thanks to Prof. Erkki Oja it is well known that a neuron using simple Hebbian Learning (including a weight decay) learns to extract the first principal component of the input data.
However, I'd like to validate my intuitions about how this generalizes to a Hebbian network which makes use of competitive learning/lateral inhibition between neurons within a layer.
So, considering I have a competitive neural network model with multiple Hebbian neurons (arranged in a layer) I would assume that the neurons roughly learn to differentiate along the first principal component.
Could anybody please validate or reject this supposition or/and provide any literature regarding that topic? Most sources only consider single or chained (Sanger's rule) Hebbian neurons.
.
maybe the following :
Dynamics of learning in linear feature-discovery networks
Tood K. Leen
Network: Computation in Neural Systems,2:1,85 — 105
http://www.ohsu.edu/xd/education/schools/school-of-medicine/departments/basic-science-departments/biomedical-engineering/people/upload/leenNetwork91.pdf
.
i am sure your supposition is correct, first time linkers in 1986 applied that idea.
i referred you to the chapter 8 of the book "Neural Networks and Learning Machines (3rd edition)" by Haykin.
"The model combines aspects of Hebb-like synaptic modification with competitive and
cooperative learning in such a way that the network’s outputs optimally discriminate
among an ensemble of inputs, with the self-organized learning proceeding on a layer-bylayer basis. That is, the learning process permits the self-organized feature-analyzing
properties of each layer to develop fully before proceeding to the next layer; this form
of learning is an example of learning features of features."
Thanks a lot for that answer, Amir. I don't have access to this book, but discriminating optimally among an ensemble of inputs sounds pretty much like what I expected :)
- Badges
- Science topic