The usual way to obtain neural networks that recognize given data is incremental training by backpropagation. But there are much better alternatives. The following is taken from section 3 of the article ''Equivalence of Perceptrons and Polyhedrons'' that I just published in ResearchGate.
BEGIN EXCERPT
Summing up, polyhedrons and perceptron neural networks are equivalent. P=PNN. So what?
Perceptron networks are often used for pattern recognition. We prefer to talk a about data recognition. The data are finite subsets of $\R^m$. A way to recognize data, much more efficient and controllable than backpropagation or support vector machines, is to construct polyhedrons adapted to the data, including specification of distances to the hyperplanes f_i(x)=0 known as ''decision boundaries''. And then convert from polyhedrons to perceptron networks. These networks recognize the data perfectly. The network are built with as much room for generalization as the data could possibly allow.
As a consequence the whole learning paradigm disappears. There is no more learning. Networks are just gestated and, as in some myths, born with knowledge. If it is the case that streams of new data keep coming, permanent online gestation will keep the network updated. These developments could also lead to other types of neural networks, quite more informative than perceptrons. It remains to explain how to obtain and combine hyperplanes that separate given data in a convenient way. There are several efficient ways to calculate these hyperplanes.
END EXCERPT
Levenberg-Marquard (lm) algorithm performs better in many cases. It is a conjugate gradient method while backpropagation is simple gradient based.Conjugate gradient has more potential to converge for an optimal solution that the simple gradient.
Cascade correlation neural network can be the candidate to topple back propagation. :)
Hello Farouk, how are you?
My experience with journals, conference proceedings, etc. is that unless you have personal or group connections with editors they will not publish your work. I have no connection at all with jcva.
One of the reasons, I believe, to launch initiatives like ResearchGate was to compensate for the biased rejection of papers.
If you are a member of that journal board, or have good enough connections there, and understand or like my paper, and can publish it there, please do not hesitate and make yourself happy.
As for the paper, it is rather technical with clear mathematical definitions demonstrations, etc.
I hope it may help ---besides providing a new technique to obtain architectures and weights--- to establish a sound mathematical theory of perceptron neural networks. I appreciate any criticism of its contents, ideas, formulas, demonstrations, viewpoints, applicability, purpose or any other relevant aspect, from you or from anyone else.
I am also grateful for the kindness of your comment, that does not mention any such criticism, but I interpret as generous advice.
Most cordially and with best regards,
Daniel Crespin
According to the no free lunch theory, We cannot claim a better algorithm for all data sets. In other words, no supper algorithm for all data sets. We can answer this question for a particular data set or a collection of data sets which shares the same characteristics. For example, we can answer this question for data sets with limited overlapping between classes or heavy overlapping. The answer could be different because the characteristics of the data sets are different.
Hi Daniel, hi all,
Daniel, I read with great attention your post that initiated this thread and I must say that my first and visceral reaction was "why the hell would one want to get rid of the learning phase?". See, my background is in psychology and more specifically in the psychology of learning and memory. And in that area the interesting part of a "perceptron" (we are rather interested in multi-layered networks, sometimes incorrectly called "multi-layered perceptrons") is that it is a model of how learning occurs.
I am not sure of what your question was or of what you propose. However, just in case what you propose is aimed at avoiding catastrophic forgetting, I suggest you read these papers :
Ans, B., & Rousset, S. (1997). Avoiding catastrophic forgetting by coupling two reverberating neural networks. Comptes-Rendus de l'Académie des Sciences, Série III, 320, 989-997.
Ans, B., & Rousset, S. (2000). Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting. Connection science, 12, 1-19.
Ans, B., Rousset, S., French, R.M., & Musca, S . (2004). Self-refreshing memory in artificial neural networks: Learning temporal sequences without catastrophic forgetting. Connection Science , 16, 71-99.
Ans, B. (2004). Sequential learning in distributed neural networks without catastrophic forgetting: A single and realistic self-refreshing memory can do it. Neural Information Processing - Letters and Reviews, 4, 27-32.
And sorry if this is off topic.
Cheers,
SCM
Hi Daniel, hi all,
Daniel, I read with great attention your post that initiated this thread and I must say that my first and visceral reaction was "why the hell would one want to get rid of the learning phase?". See, my background is in psychology and more specifically in the psychology of learning and memory. And in that area the interesting part of a "perceptron" (we are rather interested in multi-layered networks, sometimes incorrectly called "multi-layered perceptrons") is that it is a model of how learning occurs.
I am not sure of what your question was or of what you propose. However, just in case what you propose is aimed at avoiding catastrophic forgetting, I suggest you read these papers :
Ans, B., & Rousset, S. (1997). Avoiding catastrophic forgetting by coupling two reverberating neural networks. Comptes-Rendus de l'Académie des Sciences, Série III, 320, 989-997.
Ans, B., & Rousset, S. (2000). Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting. Connection science, 12, 1-19.
Ans, B., Rousset, S., French, R.M., & Musca, S . (2004). Self-refreshing memory in artificial neural networks: Learning temporal sequences without catastrophic forgetting. Connection Science , 16, 71-99.
Ans, B. (2004). Sequential learning in distributed neural networks without catastrophic forgetting: A single and realistic self-refreshing memory can do it. Neural Information Processing - Letters and Reviews, 4, 27-32.
And sorry if this is off topic.
Cheers,
SCM
LM is a better algorithm. Also Scaled Conjugate Gradient is a good algorithm.
There are varieties of backpropagation that works much faster than the classical backprop: resilient backprop. But all depends on data.
Hello Serban
My posting is about avoiding all forms of incremental learning, as are backpropagation [BP] and other gradient methods that attempt to minimize error functions. In other words, I want to avoid iterative methods or processes that rely on repeated attempts where each attempt modifies the previous one aiming at small improvements obtained via a mechanism acting in reverse direction along the neural network.
To the best of my knowledge, no biological mechanism similar to the BP algorithm has ever been identified. The currently popular algorithm of backpropagatin an infinitesimal error along the neural network has no known neurophysiological or molecular counterpart. Therefore the choice of BP and its variants as a formal model of learning could be a risky bet.
As a general viewpoint I assume that the basic biological neural mechanism is "discernment". In minimalistic version this means that
1.- Biological neurons are capable of discerning at least two different situations and react differently to these in a way detectable by other neurons. This is perfectly acceptable and hardly under discussion.
2.- A formal model that mathematically reflects
a) discernment by individual neurons and
b) interconnections or way individual neurons affect each other.
This should suffice for in many situations where understanding behavior is the goal.
The collective response of large numbers of interconnected neurons ---each individual neuron conceived as a discerning unit--- would be what we call behavior, including learning.
These ideas are not new. They are much older than BP and remain accepted in the neural literature, but have perhaps been distortioned, confused and beclouded since the advent of BP. My posting is about highlighting direct construction and downplaying BP.
See my papers on the mathematics of NNs, available at
http://ucv.academia.edu/DanielCrespin
Particularly "A Primer on Perceptrons", "Generalized Backpropagation" and "Equivalence of Polyhedrons and Perceptrons". More details will be provided in the not so far future.
Nevertheless, I have not found ---nor have made a great search effort for--- biological mechanisms similar to the proposed "direct calculation of architecture and weights". Nevertheless, the procedure is much more efficient, natural and controllable than BP.
A far away lighthouse for the search of a biological mechanism correlated to the direct construction algorithm could be the important work of Eric Kandel on Aplysia californica. But I consider this is only an "Aplysia Lighthouse Conjecture" that remains below my actual horizon.
Serban, I am thankful for your time and interest.
WIth most cordial regards,
Daniel Crespin
- Badges
- Science topic