Neural Networks very loosely imitate and are inspired by the human brain and as such have some the strengths of the human brain like discrimination/pattern recognition and some of the weaknesses like forgetting and not being able to multiply 2n digits efficient
y. Neural Networks are used in Machine Learning a subset of AI and in AI itself in many different applications.
The direct answer is this: in any area of AI, from the very beginning, the unfortunate 'tradition' has been not to criticize any popular models in order not to undermines the 'reputation' of the area. The reason for this 'tradition' (related to the need for funding) has to do with the fact that so far none of the existing models came even close to adequately capturing any important facet of intelligent information processing.
ANNs are advanced function approximation machines and any resemblance to the brain is quite superficial. From our collective experience, it is not that difficult to conclude that brains are not approximation machines (hence the effects of over-training in ANNs) .
Krzysztof thank you for your reply. I wanted to clarify something. In many cases having much data can be a good thing, and yes proper weighting/parameterization are important factors too. My concern is the lack of reporting and analysis of how too much training data interferes with generalization.
Lev, thanks for your response and I tend to agree with you based upon my own experiences, the peer reviewed literature and talking this out elsewhere. I think ANNs are very useful in proper context. I do have one disagreement, I do think there is intelligent analysis of patterns that humans cannot perform, nor can linear non-neural based programs as well. For example, in bioinformatics we have enhanced NN analysis of DNA/Protein micro-array data via clustering from ML and AI techniques. I do, however, thank you for your well thought out and useful response.
Jacob, please note that I have not said NN are not "useful". I said "that so far none of the existing models came even close to adequately capturing any important facet of *intelligent* information processing". We have much *useful* software, but that does not mean that it is "intelligent" software. This confusion has had an extremely negative effect on the development of AI.
(see my review of Nilsson's book
http://www.amazon.com/Quest-Artificial-Intelligence-Nils-Nilsson/product-reviews/0521122937/ref=cm_cr_dp_qt_hist_three?ie=UTF8&filterBy=addThreeStar&showViewpoints=0
and especially my developing book http://www.cs.unb.ca/~goldfarb/BOOK.pdf )
Lev, thank you. I read your review at Amazon and I also began reading your book in progress: Redoing Our Mathematics and Science. Please also note that I stated not only is AI useful to which we agree, but also that it also employs intelligent analysis. I define "intelligent" as the ability to adapt to new circumstances via learning and thus, identify novel patterns, all of which ML and AI meets, in criteria. Now, we do both agree there is not enough critique of the "progress" of AI/ML, but at the same time new gene expression patterns and disease pathology paths, have been elucidated using AI/ML techniques. From my study of neural nets for example it can be seen that they grew in disfavor in the late 1990's due to: high financial/computational costs and realizations of limitations like in the case of perceptrons earlier on as well. Now, NN and other ML applications, subsets of AI are back in full swing as a result of better computational efficiency and reduced cost monetarily. However, generalization limitations even with proper weighting and high throughputs (pipelining) there are limits to generalization that goes almost completely unreported.
Jacob, after more than three decades of research in pattern recognition, I gradually came to the conclusion that the true progress in AI will not be as incremental as has been assumed. In other words, once we unlock the main, foundational, door we will be well on our way towards the development of AI.
The reason has to do with the historical and pivotal role of AI in science in general. As I discuss in section 1.2 of my developing book (ROMAS), AI for the first time in the history of science became directly involved in modeling the mind. The fathers of the Scientific Revolution of mainly 16th --17th centuries (Descartes, Newton, Leibniz, and others) have explicitly excluded mind from our scientific picture because they realized that, in contrast to their spatial foundation of physics (and mathematics), intelligent information processing is of non-spatial nature. This means in particular that we cannot rely on the conventional scientific formalisms when addressing AI, since they evolved based on the ancient (spatial) measurement practices going back more than 5000 years ago.
We (together with the fathers of the Scientific Revolution) should understand this transition as much more radical than science has experienced so far. So the true opening of AI cannot be a matter of fact scientific affair and will have profound and somewhat obvious implications to all science. It appears that the only candidate to produce such effect can be a fundamentally new, non-numeric, representational formalism. We have proposed one such candidate formalism (ETS).
So my main point regarding the development of AI (and PR/ML) is that there are inherent limitations of the numeric representations which cannot be overcome by new clever theories. Specifically, the concept of class representation is not meaningful within the numeric formalisms.
Descarte's cogito does not exclude mind. I would also say that modern definitions of AI means intelligent behavior more than intelligent thought.
Now, Descarte's distinction of math is a reality is a quite fascinating discussion.
It really depends on what type of data you may use. As with any formal system, if you provide more and more for a system to use to construct its analysis, the more likely you can miss outlier cases (in more general cases). Using 'too much' training data can really be the case of not using the right parameters, and maybe using too much of the same type of data. For instance, I wouldn't just feed an ANN the exact same looking data for everything, balance out the cases so that less likely cases it may try to catch can be 'roughly' considered. It is a flaw of any ANN is the input given, and the problem domain it can be applied to. Since ANN can consider a lot, but should be used carefully when it comes to lots of different situations. Independence of problems can be a big part of this. This is not discussed possibly as much due to just people not understanding the limitations of ANN. People think of them when they don't understand them as well as 'the greatest thing since sliced bread', when they don't yield much more power than some other methods, much like any field of algorithmic research. People find their power in expert systems, and predicting input such as handwriting especially. Usually people get in trouble when they expect them to be like 'us', when computation is not us or our brains. Computation is by computational models, and if we are even to consider that we would have a lot of work ahead of ourselves before we can say more but it looks a little unlikely due to the limitations even of approximations in formal systems. This is why it is not discussed amongst experts as much since they understand this while maybe non-computer science audiences may see it as this 'glorious' thing. Machine learning is a very fascinating field, but it is not best thought of as this 'human brain' replication concept, which is not what scientists should be after since we know mathematics has limitations.
Daniel thank you so much for your reply. It was thorough, humble and humbling. Yes, it is true the correct number of features and parameters with proper weights and weighting adjustments must be used. One example that comes to mind are micro-arrays (DNA/Protein) where there are few samples N, but high dimensions within each sample. This is an example of using pipe-lining of material on a biochip which yields a high throughput. With so much data AI and ML/Data Mining algorithms are required. Now, no matter how well the parameters are set, and even as a general rule we can add more training data to make it a more efficient algorithm, with modulating features number up and down, eventually accurate generalization will be reduced and the processing will become very slow.
Hi Roman, I was using Perceptrons as an example. In reality Perceptrons are v the precursors of simplified Neural Networks which were first created in the early 1960's making improvements upon the McCullough and Pitts neuron based upon Donald Hebb's work in 1949, which came to be known as Hebb's rule where AI began to be more closely modeled after real biological neurons, though, still in a loose sense. However, Frank Rosenblatt did revolutionize the path to create neural networks starting around 1962, which were developed in the 1980's.
Overfitting and underfitting are real issues to be sure, however, it is still my statement that at some point too much training data is in fact just too much.
In my humble opinion, one trouble with too much data is that, for example in time series, the nature of the system under modeling changes along time. That is, the best model of January's data is not the best model for March's data , and neither is the best overall model for all 1st quarter data. If you test your model with prediction of April, good luck! But even modeling March with the model obtained by training of 1Q data it is no better than the model obtained from training only using March data.
Neural networks themselves are non-parametric not assuming a normal/Gaussian distribution, and they can be an excellent choice for classification, and prediction modeling but they still have inherent limitations presented by having too many features/high dimensionality and at times too much training data or test data that fits well to current training sets but do not generalize to future training sets.
ftp://stat.duke.edu/pub/WorkingPapers/00-18.pdf
Neural network creates the model that need two types of processing, First one Training a neurons and find the solution and build the model. Second, Model helps to test the data remaining dataset. Training needs a lot of time.
"Too much training" indicates a fundamental error. Usual error is to use too many weights. Too many weights will be useless ourside training area (for new data), and usually will reduce performance. Reduce weights until overtraining does not occur. How to improve performance with few weights? Certain neural networks using models can learn fast with few weights.
Lev,
Great to hear from you. let's take our discussion outside this forum? my email:
- Badges
- Science topic