In a Hopfield network, all neurons are interconnected.

In a real brain this can only be true in small volumina.

It is conceivable that there are long range connection,

but they should be rare. On a larger scale, the brain

connections are those of a 3D grid.

http://www.newscientist.com/article/dn21647-human-brain-organised-like-a-3d-new-york-city-grid.html

http://www.sciencemag.org/content/335/6076/1628

A problem with autoassociative networks where the

neurons are not completely interconnected is that

different sub-networks can assume states such that

the combined state never has been learned. This can

be demonstrated with a thought experiment: imagine

two Hopfield networks that are only sparsely connected.

It is obvious that they can retrieve separate patterns

more or less independently.

Regards,

Joachim

May you please allow me to share some of my views please? (although I am an amateur in the field of Artificial Intelligence, I have many programming experiences and I have a backgound as a neuroscientist)

In regard to the "meta-forms" (if I perceive correctly the meaning):

Neural physiological functions and/or cognitions are regulated by a delicate but intricate network of different neural cells in the CNS. In this network, the way of communications between different brain regions is the main factor. Within each of these brain regions, the way of communications between different types of cells (including both neural and non-neural cells) is the main factor. At the cellular and molecular level, the way of communications between these different cells is further determined by the way of connections, types, numbers and stability of the synapses from both neurons and neuroglia. Additionally, the pattern of the synpases and the synaptic network is influenced by the physiology of these neural cells which is further determined by the expression pattern and the presence of cytokines, local factors and intra-and-extracellular signal transduction molecules by different types of cells and in the environment, respectively.

In short, understanding the mathematical pattern of these synaptic communications is the basis for the development of different algorithms used in a virtual neural network. Thus, I would guess synaptic communication pattern is the meta-form of neural circuits based on which where complex neural networks are built.

Any comments and criticisms, please?

I agree with Ping Kwan, that synaptic communication patterns (and by it corollary, synaptic efficacy) are at least part of the answer here. A very large role is also played by the time dependent nature of these communication patterns; something abstracted out of simple neural models.

Out of curiosity, why would a Hopfield network be closer to sensory neurons? Does its recurrent nature not bring it closer to association neurons? I can see how SLP's would be closer to sensory neurons (with perhaps our retinal horizontal cells a perfect example), but I fear I do not understand your reasoning here.

In a Hopfield network, all neurons are interconnected.

In a real brain this can only be true in small volumina.

It is conceivable that there are long range connection,

but they should be rare. On a larger scale, the brain

connections are those of a 3D grid.

http://www.newscientist.com/article/dn21647-human-brain-organised-like-a-3d-new-york-city-grid.html

http://www.sciencemag.org/content/335/6076/1628

A problem with autoassociative networks where the

neurons are not completely interconnected is that

different sub-networks can assume states such that

the combined state never has been learned. This can

be demonstrated with a thought experiment: imagine

two Hopfield networks that are only sparsely connected.

It is obvious that they can retrieve separate patterns

more or less independently.

Regards,

Joachim

I once gave a paper on my Neural-network research for the Navy (NN, sounds better than ANN), where a group of Neurologist were in the audience; They said "are you trying to tell us that you know something about how neuron cells work in the human brain to process sensory data?" I said, " Heavens no, I am just using a nonlinear transfer function (a sigmoid) with a NN of layered processors that performs a Feature Vector (based on extracting features from the data example with proper segmentation) classification, once it has been trained with apriori examples (i.e., exemplars)." Please see the early paper of Lippman for a good introduction.

The reality is that there are really only three types of classifiers using 1) a statistical Probability Distribution Function (PDF), with the dimensionality of the feature vector being the statistical parameters, where one is the signal and the others are for the many different types of non-signals we call clutter, and a decision is made by comparing distances from the PDF's in the feature space between the test vector and the exemplars from the training, or 2) a Compound or Vector Quantization (VQ) classifier, where decision surfaces are constructed from the same spatial sized, 'hyperspace cell," to decide when a test vector is on one side or the other of the surface, and 3) an Optimized NN Decision Surface of varying hyper-space cell size to fit all of the apriori vectors in the proper classes (e.g., Back Propagation, Radial Basis Functions, etc.). There is a large trade craft to make NN work, and it is very easy to make them not work, but the NN will achieve the best clutter rejection within a signal detection of all classifier types. Thus, one can assume that NN are inspired by human neuron knowledge, but even today, there are very few neurons in current research analysis, and most NN problems work in feature spaces of N=100 or more, with at least two hidden layers, in order to have places to "tuck away the clutter," in order to "see" the signals. Good luck

In my humble opinion, I think that the connectivity patterns determine the processing capabilities of the network, and for instance, its behavior. In general the ANN's use the McCulloch & Pitts neuron model (1943) as the main building block. Even though this model is an oversimplification of the biological neuron, it allows to study the computational capabilities of the whole network, as well as some emergent properties that we could attribute to the "sum" of all the non-linearities that compose the network, which in turn is given by the connectivity patterns.

Instead of separating neurons into :"association, sensory, and motor neurons" I separate them by function into Memory, Computational, and Transport neurons all neurons have the capability of all three of these functions to some extent but some of them are predominated by one or the other function. You can tell which one predominates by the shape of the neuron.

Memory neurons have more dendrites coming out of the soma, computational neurons have more branching in the dendrites, and transport neurons are long and thin and have few dendrite branches.

forgive me for not taking the time to look it up, but I believe Andrew Ng has referenced in his standard NN talks research showing that the edge detecting autoencoders, and phenome detecting autoencoders that appear when training ANNs are very similar to what is found in biological NNs when you for example, show different autoencoder-like images to rats and try by trial an error to find the images that make biological "hidden layer" neurons react at their maximum. This suggests that ANNs do in fact have a biological equivalent, though probably only at the level of the brain's "preprocessing" of stimuli

http://deeplearning.stanford.edu/wiki/images/thumb/3/3e/ExampleSparseAutoencoderWeights.png/400px-ExampleSparseAutoencoderWeights.png

Yingxu Wang: " What are the meta-forms of neural circuits where complex neural networks are built based on them?"

I'm not sure what is meant by the "meta-forms of neural circuits", but the synaptic matrix mechanism described in Chapter 3 of *The Cognitive Brain* can learn complex sensory patterns and recall images of these patterns. For example, see "Sparse Coding of Faces in a Neuronal Model ..." and "Self-Directed Learning in a Complex Environment". These can be viewed on my Research Gate page.

Believe or not, this fundamental problem has been largely solved, as explained in my recent (Nov. 2012) book: Natural and Artificial Intelligence: Introduction to Computational Brain-Mind. http://www.brain-mind-institute.org/ (1) Your classification of neurons is bad for modeling. Instead of sensory, associative and motor, there are two basic types of neurons, projection neurons and interneurons (Kandel, Schwartz, and Jessell, Principles of Neural Science, 4th edition, p390-331). (2) Hopfield Networks are too primitive for modeling since they do not model interneurons. (3) More importantly, a brain-like network must learn abstract logic, among many other functions like vision, audition, touch, motor, thinking and decision making. All my above points have been explained in my above book. If you have any other questions, send me an email.

One of the problems with presenting neurons as only two types, is that it doesn't allow for processing of the signal. In my work interneurons are merely small neurons with low transport function, what is also important is the computational neurons such as the inhibitory neurons that act as barriers between columns etc. These computational neurons define the heterogeneous neural groups that make up the circuits of the mind.

Dear Dr.Wang .. Thank you for your question. A good discussion has emerged because of it.

There should be no doubt that artificial NN models remain gross approximations to how biological NN work. Clearly, it is important for computational learning to stay abreast of the latest findings (at least at the 500 foot level) from neuroscience and vice versa.

This having been said, recent work in Deep Learning Architectures (DLA) by Andrew NG Stanford) , Geoff Hinton (Toronto) and Yoshua Bengio (McGill) is beginning to show that variants of early theories of Hebb (neuroscience) and Hopfield (computational learning) were good foundations for exploration. Let me explain. Hinton and Sejnowski took the concepts of Hebb and Hopfield and overlaid it with simulated annealing to give us the Boltzman Machine learning algorithm. Over he last 15 years this has led to the Restricted Bolztman Machine (RBM) algorithm and the concept of a layered approach to learning increasingly more abstract features from sensory inputs. These features are developed using an unsupervised associative learning algorithm (contrastive divergence which approximates the BM algorithm) applied to a more biologically realistic network constructs (layers only interact with proximal layers). Once learned, these features can be used to developed the best classifiers todate for problems such as hand-written digit recognition (Hinton) and object classification in images (Google and Ng). Perhaps most importantly, the work on DLAs using different modalities of sensory input suggest that supervised classification is actually just a human explanation for a complex unsupervised associative recall process that maps data from one part (modality) of the nervous system to another. The sound of the letter "A" from the auditory cortex can cause the visual cortex to generate the image "A", and vice versa. SImilary, we sense the motor neuron activity that would allow us to vocalize the sound of "A". This may be what we identify with the concept of "knowing" the letter "A".

For a recent paper that works toward a system that uses RBMs to do this type of unsupervised associative learning across multiple sensory modalities, please see the attached paper.

Dear Drs. Smith and Silver: You two raised a few next-level finer-grain issues that I did not address in my above brief top-level comments. I try to put concisely in (A) and (B) below.

(A) Finer neuronal types: There are at least a few dozens of neuron types in a brain if one finely classifies biological neurons. But, at the top-level, for modelers it is insightful to classify neurons into only two types as Kandel, Schwartz, and Jessell did and I supported in my Natural and Artificial Intelligence (NAI) book: projection neurons (excitatory) as feature detectors and interneurons (inhibitory) resulting in sparse coding through competition. Until recently, the neural network community missed a key mechanism: large-scale inhibitory competition: Researchers model single neuron neurondynamics (e.g., as Eugene Izhikevich and many others did) but they did not understand that interneurons working with projection neurons enable projection neurons to compete in large scales, resulting in sparse coding! Sparse coding (see

David Field but our account of causality is different here) is necessary because those neurons that do not fire at the current moment correspond to long-term memory which many neural network models lack (including Hopfield nets, Restricted Bolztman Machine, Deep Learning Architectures by Geoffrey Hinton, Convolutional Networks by Yann LeCun and Deep Learning Networks by Andrew Ng). In other words, all such prior networks lack an effective long-term memory mechanism. That is also why Independent Component Analysis (ICA) lacks long-term memory but our biologically inspired Lobe Component Analysis (LCA) has both optimality and long-term memory. See: J. Weng and M. Luciw, "Dually Optimal Neuronal Layers: Lobe Component Analysis," IEEE Transactions on Autonomous Mental Development, vol. 1, no. 1, pp. 68-85, 2009. http://www.cse.msu.edu/%7Eweng/research/LCA-IEEE.pdf.

(B) Until recently, the neural network community did not understand what a brain does (i.e., the brain problem). For example, I respect Geoffrey Hinton and Terry Sejnowski but I had a face-to-face debate with Sejnowski during IJCNN 2013 Aug. 7 (6pm - 7:30pm) about the BRAIN Project. They did not address the brain problem, only mainly pattern recognition problems in a clean background. For example, their models do not address the concept learning issue (e.g., learn concepts such as where, what, scale, etc.) directly from cluttered backgrounds (e.g., natural images with multiple figures under a cluttered ground) --- only the "what" problem, not the "where" problem, not the "scale" problem. Our brain-mind model (Developmental Network DN), with its embodiments Where-What Networks address, was intended to address almost the entire brain problem. That is, not only pattern recognition in a clean background or (whole) image classification, but also the general vision, general audition, bottom-up and top-down attention, cognition, abstraction, reasoning in logic, language acquisition and language understanding, and motivation. In particular, I was probably the first person who started a dynamic deep learning architecture (Cresceptron, IJCV) jointly with Thomas S. Huang and Narendra Ahuja since 1990. However, now I understand that deep learning is secondary (later and not primary) in a brain but shallow learning is primary (early and primary) in the brain. Please see why in the article "IEEE TPAMI Banned Neural Network Approaches" (http://www.brain-mind-magazine.org/read.php?file=BMM-V1-N2-paper3-PAMI.pdf).

The basic reason is that a shallow-learning architecture must be developed early in life and work before additional learning circuits can grow from the working shallow-learning architecture, resulting in a complex network of shallow and deep learning circuits. However, the shallow learning architecture is still primary and the deep learning architecture is add-on and secondary.

See "Sparse Coding of Faces in a Neuronal Model: Interpreting Cell-Population Response in Object Recognition" on my Research Gate page. The synaptic matrix has both excitatory and inhibitory neurons to achieve sparse coding.

I would add to Daniel's excellent post that it is not just "gross approximations to how biological NN work" but that they have to be put in the historical context and what phenomena they were trying to model at that time. Since then this model has been superseded by other models. In terms of computational models they have been replaced by Restricted Boltzmann machines which give way to one type of deep learning architecture.While they may provide really gross approximations of real NN their focus in benchmarks are more towards optimization results rather than biological plausibility benchmarks(though they have some plausibility). I would look into extending other recent models architectures such as:

Artificial Astrocytes Improve Neural Network Performance by Porto-Pazos et al.

Still it all depends on the level of modeling desired whether such architectures are suitable or not.

I think that there is a difference between lateral inhibition and computative neurons that define locality for things like the column and mini-column. One of the problems with ANN techniques is the fact that because all neurons are simulated with the same algorithms, there is no difference in how different types of neurons are implemented.

If we ever want to understand the structure of the laminae in the cortex we have to get past assumption that all neurons are equal and start looking at why some neurons are designed differently from other neurons. I think that my characterization of neurons as memory, computational, and transport neurons is a first step in this regard.

When our neuron models begin to reflect the true complexity of the brain, we will begin to have a much better understanding of how it works.

The precursor was an Ising Modell of ferromagnetics.

I would like to be a little provocative: All the dualities (human/artificial etc) in the attempts to build models of neurons functions with the intention to reflect them in artificial systems are nothing else then scratching the surface of the issue.

Why that? Because all cited scholars don't ask the self-evident question (also not the honorable Mr wang). What if the neuron treated as a Black-Box is not as a Whole responsible for conscious processes (Input-Output Models), what if inside neurons complex events are the real cause for consciousness occurring? Firing is only the result...

What if non-computational processes on microtubules level inside neurons, quantum states, quantum coherent superposition in self organizing processes as preconscious phase collapses in discrete conscious events, are the real explanation for consciousness ?

What if since Goedel scientist have to deal with the fact that classic physics and logic are only a restricted level of rational duality that works well in its boundaries until we are confronted with quantum physics and quantum logic?

All this questions are a part of a deeper attempt to cross all those old boundaries of conventional thinking, cleaning our habitual thinking load and recognizing that conscious experience is intimately connected with the non-classic physics underlying

space-time structure, pre-logic, quantum theory - and - microtubules. (see also Penrose/Hameroff publications to this subject)

Evolution is not a mystical event. Life and consciousness as results of an evolutionary process may appear mysterious and enigmatic. Quantum physics is so incomprehensible that some scientists hoping to find the Holy Gral in this dark and mysterious world. So some of us take refuge to quantum states, non-computational processes on microtubules and other high level ideas - all heavy guns to explain these phenomenons.

For me it is very serious, honorable and proper science.

Nevertheless I feel a smell of homunculus or Maxwells demon in the scull about such concepts.

I am sure, consciousness needs no scientific ghost but a clever idea of data processing to explain this phenomenon. I am also sure, we will find it in a set of anticipation, association and feed back loops, clever connected and related - all simple and isolated well understood.

Dear Prof. Musterle, I am aware about the history of the Tuebingen University especially since I was one of Ernst Bloch students. Modern revolutionary Philosophy, new way of thinking, changing economic and social developments and relationships, all those produced new ideas and opens our mind.

Since than Things happened. From classic system theory and cybernetics we went forward and deeper toward questions about our macro and micro universe.

We understood that one of the most complex product of evolution is our brain. There is a huge amount of research about his function. We all recognized that it is a fundamental difference between classic computation, artificial systems and neuronal complexity of our brain.

All "Stand der Wissenschaften", rejected, I read your statement and publications about natural and artificial systems that "The behavior of such systems is in any occurring situation a simple stimulus-response-reaction"

(https://www.researchgate.net/profile/Wilfried_Musterle/)

and asking myself how comes this kind of ignorance ?

Quantum theory, quantum logic, has nothing but nothing to do with "mysterious". We are not so far yet to understand all nature evolutionary processes, but we are not dealing with "ghosts". and surly conscious is not simple dealing with data. This last statement will throw us for decades backwards.

I understand your wish to simplified things, but nature and science confront us all the time with new phenomena that are not to explain with simple "feedback loops"...

And now down to business. Personally I think that Penrose/Hameroff has taken us a giant step forward threw opening a discussion on two levels. Are conscious events computable? Are neurons firing cause and product of cognition and volition?

The moment I red about the inner process of neurons and how Microtubules are acting, it was clear for me that there is a strong connection between Quantum logic as pre-logic and the quantum state, the superposition of tubules. In fact it fits like glove on hand.

To understand brain function we need an adequate logical framework. We see that our classic logic is not able to describe even the process of the amoeba split without contradictions. There are some total radically new ideas about how to formalize thinking about thinking but the resistance against this new in the conservative science community is yet strong and disaffirm.

Ilan,

thank you for your kind respond on my provocative post. I agree in most of your opinions. If you read my last post, you can see my appreciation of your research. Nevertheless I am sure that quantum physics is not the fitting tool to understand consciousness.

The cellular chemical reactions are not processes with single ions, molecules or subparticles but some billions or some more of them, so are the results of such reactions the statistical sum of all interactions. In this statistical sum all quantum effects will disappear. Quantum effects are very interesting to understand individual interactions of molecules in chemical reactions, but not for the result of the reaction in common.

Consciousness is also not the result of any singular interaction of atomic subparticles or single molecules. It is a phenomenon in the whole brain an the activity of very much neurons. If quantum effects would cause consciousness it would be more randomly and not stable as we find. So we should ask different questions as physicians even if these questions for the basics of our world would be more exciting.

Now, you may understand a little, why I think quantum physics is not fitting "like glove on hand." I also refuse not any of the new ideas nor I think that the classical logic collapse. In my wish to simplify things I regret your missunderständing of my little publication but there is more than only the introduction. Last but not least, I am not a Professor.

Wilfried,

Classical physics is an ancient way of thinking in physics which has been replaced new type of physical thinking in the last century. Unfortunately the new way of thinking about the physical world is not known by non physicists. All other scientists and engineers are trained in the mode of thinking of classical physics. Most non-physicist scientists think that since there are dealing with macroscopic phenomena, they should not care about the physics of the quantum. Maybe they are right but maybe what is most important in the new physics are not the equations that applied to a microscopic world but the new mode of thinking of the new physics. Gennaro Auletta is drawing a few parallels between general aspects of quantum systems and biological systems in this paper:

Organisms as a New Class of Physical Systems

https://www.researchgate.net/publication/236617071_Organisms_as_a_New_Class_of_Physical_Systems?ev=prf_pub

Chapter Organisms as a New Class of Physical Systems

Louis,

I have studied physics and chemistry - I know very well quantum physics - so sorry.

Louis,

nevertheless I am sure that we need no "new" physics to understand the phenomenon of consciousness.

I just wrote: "If quantum effects would cause consciousness it would be more randomly and not stable as we find. So we should ask different questions as physicians even if these questions for the basics of our world would be more exciting."

Wilfried,

The point of my last post what not to say that we need a new physics but maybe the new mode of thinking associated with the new physics. Major new ideas, new discovery are often very simple but they are simple in a new mode of thinking.

Quantum physics is very interesting and in a special way strange. It may be, that some of us believe that they are entering in Alice wonderland because of the wondering effects. The effects are different and not compatible to classical physics.

I do not wonder. Each subject of ​​physics, including quantum mechanics, has its own scope. It does not make sense to use mechanical models in optics. Quantum mechanics has its validity in the field of atomic and subatomic dimensions. It is doubtful, to try to understand the activity of the brain with quantum mechanical models. We have statistical interactions of nerve impulses, which gain meaning through their synchronization in the brain. There is no room for quantum effects. The only quantum effect, which is at least funny in connection with the brain, is the little poem in the style of Christian Morgenstern by Werner Heisenberg:

Ein Wirkungsquant fliegt durch das Dorf,

es sucht das Hirn des Herrn von Korf.

Es findet dort in dem Gewühl

ein ganz bestimmtes Molekül.

Von Korf ist grad in schwerer Not:

„Eß' Wurst- ich oder Käsebrot?“

Das Quant, das wirft sich in die Brust:

„Du glaubst, du willst! Allein: Du mußt!

Nie kannst die Freiheit du erringen.

Doch ich bin frei und kann dich zwingen!“

Elektron „9“ sprach: „Spring' mich doch!“

Das Quant: „Ich überleg' mir's noch.“

Dann hat durch es Elektron „8“

'nen akausalen Sprung gemacht.

Von Korf nahm daraufhin spontan

die Wurst und fing zu essen an

und nahm die Sache ganz im Stillen

dann als Beweis für freien Willen.

Dem Quant hat das den Rest gegeben:

frei-willig schied es aus dem Leben.