The statement doesn't mean anything until you define "learn". For instance, the NN language you describe, in a cynic's opinion, is just curve fitting. Someone defined the form of the problem, chose the pertinent data, coded the input and the output, then declared the network "learned" the answer to my question.

A breakthrough might mean that the system can recognize a useful question and the important information required to answer the question.

Hello, James Taylor

Learning can certainly be considered curve fitting, at least in the case of perceptron neural networks (PNNs). See arguments below. A person can be a cynic, but not because he stated this truth.

Curve fitting usually refers to a collection F of real valued functions y=f(w,x), of one real variable x, that depend on parameters, say w=(w_0, ..., w_n), also called weights. For example, F could be the class of all polynomials of degree n in one variable. Note that the (n+1)-tuple w of real parameters can be viewed as a single (n+1)-vector parameter.

Parametrized collections of real valued functions of one variable determine a corresponding parametrized collection of graphs in the XY plane.

If points (x_1,y_1) ... (x_q,y_q) are given in the plane, curve fitting is any algorithm that finds a specific numerical parameter vector w such that the graph of the function x->f(w,x) contains the given points or, most often, the graph is the closest (or is a graph close enough) to the given points. Here the problem diversifies depending on the specific meaning of "close" (absolute values, sums of absolute values, sums of squares, sums of p-powers, etc.)

When dealing with real valued functions y=f(w,x) of two real variables (x is then a vector in Euclidean plane R^2) and data points (x_1,y_1) ... (x_q,y_q) (these are now points in 3-dimensional Euclidean space R^3) the above notions generalize, but "curve fitting" could be misleading, and "surface fitting" would be a more appropriated term.

More generally, in the case of real valued functions y=f(w,x) of m real variables we have that x is a vector in m-dimensional Euclidean space, the data points (x_1,y_1) ... (x_q,y_q) lie in (m+1)-dimensional Euclidean space and instead of "curve fitting" a better term would be "hypersurface fitting". This is because, in the present case, for each specific parameter vector w the graph of the function x->f(w,x) is an m-dimensional hypersurface contained in (m+1)-dimensional Euclidean space.

Recall now that the characteristic function of a subset K of R^m is the bit valued function c_K:R^m -> {1,0} that takes value 1 on K and value 0 on the complement R^m-K.

On the other hand, the so called "forward pass" of an n-input perceptron neural network P, to be denoted f_P, is also a bit valued function, f_P:r^m->{1,0}.

A theorem then states that a bit valued function of m variables g:R^m-> {1,0} is the characteristic function of some polyhedron K contained in R^m, g=c_K, if and only it is the forward pass function of some m-input multilayer perceptron neural network P, g=f_P.

Each P in the above statement determines, besides the forward pass function f_P, a unique polyhedron K_P.

Consider a set of perceptron networks parametrized by weights, their forward pass functions, the graphs of these functions and the corresponding polyhedrons and consider a given set of data vectors (x_1,b_1) ... (x_q,b_q). Here each x_j is a vector in R^m and each b_j is a bit.

Network P "perfectly knows the data", or just "knows the data" for short, if its forward pass function satisfies f_P(x_j)=b_j for all j. This is equivalent to all the data vectors being contained in the graph of the forward pass function f_P. Alternatively, P knows the data if its forward pass function f_P fits perfectly the given data vectors. To know means to fit data perfectly.

Suppose "P does not know the data", which will be also stated as "P ignores the data". This means that the data fitting is imperfect. Equivalently, that the equality f_P(x_j)=b_j fails **for at least one index j**. In the extreme case that f_P(x_j)=b_j fails **for all indexes j**, one can say that "P totally ignores the data" or that "data are totally unfit to P".

It is possible to be more specific with individual data and say "P knows data point (x,b)" if P(x)=b. Percentages of individually known data can be calculated giving sense to statements like "P succeeds in 75% of cases" or "P fails in 25% of cases".

Let P=P_{w^{ini}} be an initial PNN specified by certain initial weights w^{ini}. If P knows the data there is nothing to be done. If P ignores the data then "P learns the data" means the following: Starting from the initial network P_{w^{ini}} and the data vectors, new weights w^{knw} are calculated such that P_{w^{knw}} knows the data. An equivalent statement is this:

"Starting from P_{w^{ini}} and the data vectors, calculate new weights w^{knw} such that P_{w^{knw}} perfectly fits the data."

As expected, learning consists in passing from ignorance to perfect knowledge.

The approach to learning called backpropagation (BP) consists in obtaining from the PNN a differentiable neural network (replacement of discontinuous thresholds with sigmoids), setting up a quadratic error function, applying gradient methods to minimize error, and returning back to a discontinuous PNN. Aided by a variety of rules of thumb, tricks, methods and techniques, BP is by and large the most popular manner to go from ignorance to knowledge. Experience has shown, however, that BP has a number of limitations and rarely provides a PNN that perfectly knows the required data.

Summing up, the considerations above suggest that, with appropriated changes, PNN learning is a special case of curve fitting.

For certain problems m-dimensional space with large m is a veritable maze. For example, if a procedure requires m steps and at each step there are two choices, the total number of possible paths is 2^m. When passing from curve fitting to hypersurface fitting, similar highly non-trivial facts are involved.

When "curve fitting" means "hypersurface fitting", we are talking about complex Mathematics. And beware, m-input PNNs belong to the realm of m dimensions. If "learning is just curve fitting" was to be derisive it may fail its intended purpose.

With most cordial regards,

Daniel Crespin

"learn", "perfect knowledge" are terms for poetry and and the social sciences. "learning is just curve fitting" is your phrase, and is precisely the opposite of my proposition. The question on the table is to identify a mathematical definition of learning, so we can determine what Mr. Gates may have meant - although this may also be reading too much into his probably rhetorical statement.

Learning is not just curve fitting, M-dimensional or otherwise; weight updates to improve a pattern mapping - training the net - may well have a functional analog in a learning process, but I don't think that it subsumes the entire process. Look at the tired analogy used for neural networks, in reverse. How is our knowledge of the brain like the neural model that you are describing? Our common descriptions of the brain involves an idealized model of one tiny structure - a neuron. Ignored are the very specialized structures which preprocess audio, visual, pressure, temperature, chemical, and somatic environmental cues. Low level processing which correlate environmental signals, memory which stores and correlates past correlations. Anticipation, speculation, trial solutions, hypothesis testing, all play a role in learning in the broad human sense.

Jumping from neuron to global truth is too big a leap. Define learning mathematically, think like an engineer or a mathematician. What can a learning system *do* that a generalized mapping can not. Think proof, verification and validation - how do you show that under your definition that learning takes place?