The statement doesn't mean anything until you define "learn". For instance, the NN language you describe, in a cynic's opinion, is just curve fitting. Someone defined the form of the problem, chose the pertinent data, coded the input and the output, then declared the network "learned" the answer to my question.

A breakthrough might mean that the system can recognize a useful question and the important information required to answer the question.

Hello, James Taylor

Learning can certainly be considered curve fitting, at least in the case of perceptron neural networks (PNNs). See arguments below. A person can be a cynic, but not because he stated this truth.

Curve fitting usually refers to a collection F of real valued functions y=f(w,x), of one real variable x, that depend on parameters, say w=(w_0, ..., w_n), also called weights. For example, F could be the class of all polynomials of degree n in one variable. Note that the (n+1)-tuple w of real parameters can be viewed as a single (n+1)-vector parameter.

Parametrized collections of real valued functions of one variable determine a corresponding parametrized collection of graphs in the XY plane.

If points (x_1,y_1) ... (x_q,y_q) are given in the plane, curve fitting is any algorithm that finds a specific numerical parameter vector w such that the graph of the function x->f(w,x) contains the given points or, most often, the graph is the closest (or is a graph close enough) to the given points. Here the problem diversifies depending on the specific meaning of "close" (absolute values, sums of absolute values, sums of squares, sums of p-powers, etc.)

When dealing with real valued functions y=f(w,x) of two real variables (x is then a vector in Euclidean plane R^2) and data points (x_1,y_1) ... (x_q,y_q) (these are now points in 3-dimensional Euclidean space R^3) the above notions generalize, but "curve fitting" could be misleading, and "surface fitting" would be a more appropriated term.

More generally, in the case of real valued functions y=f(w,x) of m real variables we have that x is a vector in m-dimensional Euclidean space, the data points (x_1,y_1) ... (x_q,y_q) lie in (m+1)-dimensional Euclidean space and instead of "curve fitting" a better term would be "hypersurface fitting". This is because, in the present case, for each specific parameter vector w the graph of the function x->f(w,x) is an m-dimensional hypersurface contained in (m+1)-dimensional Euclidean space.

Recall now that the characteristic function of a subset K of R^m is the bit valued function c_K:R^m -> {1,0} that takes value 1 on K and value 0 on the complement R^m-K.

On the other hand, the so called "forward pass" of an n-input perceptron neural network P, to be denoted f_P, is also a bit valued function, f_P:r^m->{1,0}.

A theorem then states that a bit valued function of m variables g:R^m-> {1,0} is the characteristic function of some polyhedron K contained in R^m, g=c_K, if and only it is the forward pass function of some m-input multilayer perceptron neural network P, g=f_P.

Each P in the above statement determines, besides the forward pass function f_P, a unique polyhedron K_P.

Consider a set of perceptron networks parametrized by weights, their forward pass functions, the graphs of these functions and the corresponding polyhedrons and consider a given set of data vectors (x_1,b_1) ... (x_q,b_q). Here each x_j is a vector in R^m and each b_j is a bit.

Network P "perfectly knows the data", or just "knows the data" for short, if its forward pass function satisfies f_P(x_j)=b_j for all j. This is equivalent to all the data vectors being contained in the graph of the forward pass function f_P. Alternatively, P knows the data if its forward pass function f_P fits perfectly the given data vectors. To know means to fit data perfectly.

Suppose "P does not know the data", which will be also stated as "P ignores the data". This means that the data fitting is imperfect. Equivalently, that the equality f_P(x_j)=b_j fails **for at least one index j**. In the extreme case that f_P(x_j)=b_j fails **for all indexes j**, one can say that "P totally ignores the data" or that "data are totally unfit to P".

It is possible to be more specific with individual data and say "P knows data point (x,b)" if P(x)=b. Percentages of individually known data can be calculated giving sense to statements like "P succeeds in 75% of cases" or "P fails in 25% of cases".

Let P=P_{w^{ini}} be an initial PNN specified by certain initial weights w^{ini}. If P knows the data there is nothing to be done. If P ignores the data then "P learns the data" means the following: Starting from the initial network P_{w^{ini}} and the data vectors, new weights w^{knw} are calculated such that P_{w^{knw}} knows the data. An equivalent statement is this:

"Starting from P_{w^{ini}} and the data vectors, calculate new weights w^{knw} such that P_{w^{knw}} perfectly fits the data."

As expected, learning consists in passing from ignorance to perfect knowledge.

The approach to learning called backpropagation (BP) consists in obtaining from the PNN a differentiable neural network (replacement of discontinuous thresholds with sigmoids), setting up a quadratic error function, applying gradient methods to minimize error, and returning back to a discontinuous PNN. Aided by a variety of rules of thumb, tricks, methods and techniques, BP is by and large the most popular manner to go from ignorance to knowledge. Experience has shown, however, that BP has a number of limitations and rarely provides a PNN that perfectly knows the required data.

Summing up, the considerations above suggest that, with appropriated changes, PNN learning is a special case of curve fitting.

For certain problems m-dimensional space with large m is a veritable maze. For example, if a procedure requires m steps and at each step there are two choices, the total number of possible paths is 2^m. When passing from curve fitting to hypersurface fitting, similar highly non-trivial facts are involved.

When "curve fitting" means "hypersurface fitting", we are talking about complex Mathematics. And beware, m-input PNNs belong to the realm of m dimensions. If "learning is just curve fitting" was to be derisive it may fail its intended purpose.

With most cordial regards,

Daniel Crespin

"learn", "perfect knowledge" are terms for poetry and and the social sciences. "learning is just curve fitting" is your phrase, and is precisely the opposite of my proposition. The question on the table is to identify a mathematical definition of learning, so we can determine what Mr. Gates may have meant - although this may also be reading too much into his probably rhetorical statement.

Learning is not just curve fitting, M-dimensional or otherwise; weight updates to improve a pattern mapping - training the net - may well have a functional analog in a learning process, but I don't think that it subsumes the entire process. Look at the tired analogy used for neural networks, in reverse. How is our knowledge of the brain like the neural model that you are describing? Our common descriptions of the brain involves an idealized model of one tiny structure - a neuron. Ignored are the very specialized structures which preprocess audio, visual, pressure, temperature, chemical, and somatic environmental cues. Low level processing which correlate environmental signals, memory which stores and correlates past correlations. Anticipation, speculation, trial solutions, hypothesis testing, all play a role in learning in the broad human sense.

Jumping from neuron to global truth is too big a leap. Define learning mathematically, think like an engineer or a mathematician. What can a learning system *do* that a generalized mapping can not. Think proof, verification and validation - how do you show that under your definition that learning takes place?

Hello James Taylor

Thanks very much for your generous words.

-------------------

About the terms "learning" and "perfect knowledge" as used in my post:

I believe I gave a reasonable mathematical definition of "learning" and "perfect knowledge". Then, why not use other words to designate these concepts?

Mathematicians have defined "ring". It is a set with two operations, etc. A jeweler can argue that this does not conform a much older meaning of the word. Or a math rookie could try to figure out a ring of algebraic integers as something having the shape of a torus. Similar comments can be made about mathematical terms like backpropagation, injection, spray, variety, manifold, Lie group and many others.

A mathematical neologism may be welcome not because it agrees with a common meaning or with poetic figures. But because it designates a novel and useful concept.

According to Saussure, words in natural language are arbitrary and can have several meanings. The various meanings depend on context. He also discovered that once a specific meaning is accepted, it becomes fixed and is difficult to change. The same happens with mathematical concepts.

In ANNs the term "learning" is not at all new. As defined by me ---again depending on context--- it could be a little less ---or a little more--- general than usual. But nevertheless it conforms the meaning found in standard textbooks and papers.

Is the term "perfect knowledge" new? Perhaps. It is certainly easy to understand and apply. But I did not introduce it in a casual way. It results from engaging ANNs during decades. In the given context of PNNs and data, the definition seems consistent. If the proper software becomes a standard tool, "perfect knowledge" may become one of those fashionable buzzword. The software I am talking about would be one that easily calculates PNNs perfectly recognizing data, and which have also sufficient generalization capability.

I hope that with these comments no further justification is needed for the mathematical definitions ---given in my previous posting--- of "learning" and "perfect knowledge".

-------------------

About "learning is just curve fitting".

Rest sure that "learning is just curve fitting" was not taken lightly by me. It is not a definition. It is ---as you imply--- a proposition. I am not surprised it is the contrary of your proposition.

It was justified after defining "learning" and "curve fitting". If you found errors in the arguments I gave, please accept my anticipated thanks. I am anxious to apologize to you and to all other readers. But please let us know the objective technical details of my mistakes. Do not conform yourself with simply telling your subjective perception of my bad literary tastes, if that was the case.

-------------------

A reasonable definition of "global truth" is an interesting problem. But I am not familiar with enough examples that could provide an orientation to attempt such task. Certainly, it is not within my mathematical experience. May it be related to Gödel's Theorems? To local truths ---if any--- in topos categories?

-------------------

You touch other interesting topics about which I could possibly say a few things. But each has a life of its own and requires extensive treatment. They are at some distance of this posting of mine. At the moment I prefer not to wander too far away from my original question.

If in the future you send a posting with a specific enough question within my reach, it will be my pleasure to make a contribution.

Most cordially,

Daniel Crespin