Hossein, if we will listen carefully the phrase: "Machine learning for Pattern Recognition", it will stay more clearly to us. Not "vs", but "for". Thus if you use the methods of Machine learning it is not only for problem of Pattern Recognition. And if you solve the problem of Pattern Recognition it is not necessary to use only Machine learning methods.

I don't think there are any real differences between the two. The variability between different PR courses, and between different ML courses,  surely crosses by far any attempt to draw a line between the two.  This being said, I tend to relate "Machine Learning" more to "Computer Science" , and Pattern Recognition more to "Electrical Engineering", but this is a personal bias. Both are within applied math.

The main differences are not scientific but 'political'.

The main problem addressed in both of them is basically the same (classification / learning), but since this problem even now  has not been adequately addressed, in the 1980's, within AI, a group of people started ML as a separate area which was not relying much on the statistical machinery.  Later on ML included statistical formalisms and the boundary between the two completely blurred. Over the last 15-20 years ML was 'politically' more successful in attracting many young researchers and creating many peripheral research  directions not very relevant to the main problem.

Of course, you will hear all kinds of 'justifications' for the artificial separation, but this is all 'politics'.

In general, young researchers would be wise to keep in mind that since hardly any substantive progress has been made in AI, to get attention and funding all kinds of politics are being played out. See also  my Amazon review of "The Quest for Artificial Intelligence"   http://www.amazon.com/gp/review/R1PYIIY121MJOX/ref=cm_cr_pr_rvw_ttl?ASIN=0521122937

Pattern recognition does not necessarily imply learning (e.g. ICA processing implies unmixing and requires "recognizing" a pattern but this  does not necessarily fall within learning, so do other algorithms such as PCA, Kallman filters, etc.). Learning on the other hand needs recognition of the pattern to be learnt in order to carry the learning process.

I will agree with Arturo, since we can develop a method to identify a specific hidden by noise attribute (for example: the inflection point of a noisy curve, provided that we have only a set of its points - see: https://www.researchgate.net/publication/228059400 and http://arxiv.org/abs/1206.5478v2 -) and it is not necessary to force any kind of machine to learn something...

Article Developing methods for identifying the inflection point of a...

With all due respect, I can't agree with Arturo & Demetris.

Let's not put the cart in front of the horse: let's not confuse many current auxiliary techniques for data processing with the basic problem of PR. Keep in mind there are many many useful programs. In fact, practically any program is 'useful', but that does not imply its attribution to PR.

Again, the way I see the resulting confusion is that it is a consequence of the lack of an adequate approach to PR. People start claiming all kinds of things as relevant to PR.

would you mind me asking what does "Learning" stand for in Machine Learning? is it related to the "Learning" as we humans do and know? or it's about fitting something to a curve or something like that?

Hassan, this is the problem.

Because we still don't even have an adequate formulation of the basic problem, people are trying to solve other, more familiar, problems and claim that this is "ML".

For example, we are trying to do "classification" without the concept of class. I.e. we still don't have a formalism that would clarify what a class is. Ask yourself: Does it makes sense to attempt classification without your model giving a clear definition of the concept of class? 

Pattern recognition are the methods that machine can used for the recognition of patterns in different type of signals.  Macnine Learning are the methods that machines can used in order to learn doing  a task or learn improving a task.  The two topics can overlap when the task is pattern recognition and the pattern recognition method is derived from a pattern training method.

I think ML is a broader field. From my view, ML is to teach a machine to do any tasks. PC is a technique for a machine to recognize some patterns in the real world.

I think Pattern recognition is done using machine learning methodology.

If we talk about training in pattern recognition, we usually have in mind trained neural networks. But the trained neural network is one of the methods of pattern recognition. Many recognition methods are not associated with training. At the same time, machine learning is related not only and not so much with pattern recognition.

Hossein, if we will listen carefully the phrase: "Machine learning for Pattern Recognition", it will stay more clearly to us. Not "vs", but "for". Thus if you use the methods of Machine learning it is not only for problem of Pattern Recognition. And if you solve the problem of Pattern Recognition it is not necessary to use only Machine learning methods.

Lev,

I think that you should divide the problem into three parts:

1)The definition of pattern recognition vs learning. I think we can all agree that for there to be pattern recognition there does not need to be learning(programs can be coded to recognize the pattern but not have the ability to learn). Learning is an additional component where the system is equipped with the additional capability to adapt on its own to new data for improved future performance.

2) what algorithms  are considered to be part of machine learning and pattern recognition

3) The historical debate on the on scope of each of the two fields

The first point was the one that I wished to address. The other two points are very debatable and we could agree to disagree.

Hope this helps

I would say PR is a sub-category of ML: the difference being the features that are extracted. Simply put, ML involves extracting useful features from data and building a statistical model. In PR, those features usually being extracted using advanced filtering, signal processing, time-frequency analysis etc, where as other applications of ML such as Natural Language Processing might use word counts, bag-of-words, etc.

I would agree with many of the answers below. For example in hyper-heuristics, a form of machine learning, many black boxes architectures have prevented the pattern recognition so far.

Some research have used pattern recognition to produce better algorithms. We have recently suggested to open such architecture and analyse the generated algorithms to recognise pattern. First  by human activities, and more preferably by computing methods. 

Very interesting questions. Thank you.

Again, what I see here is a high level of confusion, which inevitably is the main result of the obfuscation of the central scientific problem within ML. The reason: when numerous young researchers with various backgrounds were recruited to ML, no central focus was clarified, and because many of those researchers came from CS they started to pursue algorithmic, statistical, and a wide range of other non-central issues.

However, the central problem is that of learning/classification, and this is true for all applications, including, computer vision, data mining, (semantic) search engines. Why? Because this is how all biological organisms, including us, orient themselves in their environments, i.e. interpret objects in various environments. I don't think we will be able to discover our way around this form of orientation. (In fact, I think it would be very very foolish to even attempt this, but unfortunately this is what has been happening in AI.) Of course, non-trivial classification is not possible without learning.

But to do classification properly, one needs a formalism that allows to specify the concept of class, since it is a waste of time to even attempt classification without (formal) understanding what a class is. The vector space representational formalism, for the reason of its internal (underlying) structure, does not allow for the introduction of any reasonable concept of class.

Dear Leb, please give us next two:

Thank you.

Then we will be able to discuss deeper.

There is no universally agreed definitions, and it is in fact some controversy over what to include within ML.  Take support vector machines for instance. In the part of the literature with which I am familiar, it is consistently counted as an ML algorithm.  Yet the extent of learning is no more than what you find in statistical regression (which is certainly not counted as ML).

That is different from the situation in (e.g.) neural networks where the network can learn a bit, then be used to make a decision, and then learn some more to be even better at the next decision task. There is a much stronger case for calling /that/ learning than the SVM which is either trained or not trained, and has to start from scratch to benefit from a new training set.

The closest we can get to a definition is to say that machine learning refer to relatively recent (20-30 years) advances in pattern recognition. In other words, the distinction is politically motivated, aiming to instill an illusion of novelty, as already mentioned by others.

And the ambiguity is not necessarily a problem either. Any one researcher will necessarily have to focus on one subarea of ML/PC/whatever (at a time), and remain ignorant of important areas and opportunities until way into the post-doctoral career. It is much more important to be aware of the limits of own understanding and the fact that there are other people approaching the same problem from a different angle, than it is to pin down a precise definition of what is or is not the same area.

To put ML/PC to any significant use, one has to combine fundamental understanding of statistics, algorithms, and implementation, as well as domain understanding including both abstract and concrete understanding of the applied problem. One might try to exclude one or more topics from the definition on ML and/or PC but not out of any useful application of ML and/or PC.

Robert Duin wrote an interesting article about this issue in his blog:

http://www.37steps.com/638/machine-learning-and-pattern-recognition/

Thank you dear Jain

Very useful link.

Pattern recognition consist of recognizing a pattern using a Machine Learning  (computer). For this, ML algorithms have been developed to map how patterns are modeled, classified and recognized. 

A similar question was asked in RG before:

https://www.researchgate.net/post/What_are_the_differences_between_machine_learning_pattern_recognition_and_data_mining

Very interesting Omid. Thank you. 

So what is data mining in this context? 

Patricia 

Thank you Patricia, There are similarity and slight difference between Machine Learning (ML) and data Mining (DM) methods. According to Wikipedia: 

ML focuses on prediction, based on known properties learned from the training data.

DM focuses on the discovery of (previously) unknown properties in the data. This is the analysis step of Knowledge Discovery in Databases.

Did you know that there is "International Conference on Machine Learning and Data Mining in Pattern Recognition"  every year. The 11th one will be held in July 20-23, 2015, Hamburg, Germany. The title says it all. ML, DM, statistical techniques, etc. are all being applied in PR. In fact ML and DM overlap in many ways. DM uses many ML algorithms and vice versa, ML employs DM methods as "unsupervised learning" or as a pre-processing step to improve learner accuracy. .

Source:

http://en.wikipedia.org/wiki/Machine_learning

Demetris,

Please don't ask questions as to " a formal definition of " any field, because such "definitions" do not and cannot exist. This is not a good sign. ;--)

(Even mathematicians have never attempted a formal definition of their field.)

Hans,

To some extent I agree with you. But I strongly disagree regarding:

"And the ambiguity is not necessarily a problem either."

The main problem with the emergence of ML outside PR is that this fragmented the field and the main efforts without offering any fundamentally new insights into the basic problem. And this is what creates the present, very counterproductive, confusion, which is clearly presented in these answers.

Lev, 

fair enough.

I would however, be less concerned about the new ideas being fundamentally new, than about their being /useful/. Knowing too, that my understanding is centered around selected problems and selected solutions, I would never exclude the possibility that useful novelties have resulted in a different subarea or adjacent field.

One result of the fragmentation is that more people with more diverse backgrounds have taken an interest in the field, and contributed to its proliferation. A certain level of confusion is a good sign, because it means that ideas have found applications in new areas. It takes time to show and agree that these ideas are not fundamentally new, but in fact old ideas in new wrapping. Even old ideas can make advances in new areas ...

Hans,

Here I would disagree with you even more regarding:

"I would however, be less concerned about the new ideas being fundamentally new, than about their being /useful."

As I mentioned above, practically all programs are "useful". So what?

The main problem we face is the data organization, and it is not going to be solved in the usual incremental manner, relying on the conventional forms of data representation, mainly numeric. Somehow there aren't many people who understand that this is so. I'm a mathematician by education, but working in the area of PR gradually realized that without radically new, non-numeric, formalisms for data representation we are not going to make any substantive progress in this direction. 

Lev, 

of course, if I had had  a job within basic research with the freedom to pursue the fundamental question whatever the cost, then I would agree with you.

But the fact is, there are plenty of problems which require a solution and which do not require a fundamentally new method of data organisation. From a mathematical viewpoint, they are special cases of no particular significance. From another viewpoint they may be the obstacle which blocks another breakthrough.

I very much doubt that the fragmentation and confusion is the cause of lacking interest in the fundamental question that you want to ánswer. A much more likely cause is that the resources to tackle any fundamental questions is hard to come by. The world at large much prefer to scratch the surface for some short-term gain.

So much as I object to a narrow focus on one main problem to the exclusion of others, I am interested in the `main problem' as you see it, even though I am not sure exactly what you mean. Would you like to elaborate? Where is the limit for current methods and what kind of problems will make current methods break?  TIA

Hans,

I wouldn't have been insisting on those issues, if our small group had not made some substantive progress. See popular introduction at http://www.cs.unb.ca/~goldfarb/BOOK.pdf and more technical papers mentioned there. (See also my RG page).

It means that there are more efficient ways to proceed with the development of PR/ML, but they lie outside.the present more conventional directions.

As to your question "Where is the limit for current methods", I already mentioned that it is meaningless to attempt PR/classification without having the concept of class (of 'similar' objects).

Cheers Lev,

interesting read.  I'll try to find the time to read it out.

However, I disagree that you have identified the core of the big question. Non-numeric representations are nothing new; we have plenty of examples. The challenge is not representation, but /arithmetics/ in the representational space. Representation is comparably straight forward, but without arithmetics the use is rather limited.

Furthermore, the basis for tackling such fundamental questions is to have a large and varied body of sample applications and sample solutions. That is needed to illustrate and to validate new generalisations and abstractions. Thus, the fragmention of the community and extensive work on non-central issues are not to be scorned, but rather to be praised.

In short, you discuss important questions, but most of us have to spend most of our time on less central questions with better prospects of answers in the immediate future. Yet, I for one, will spend /a little/ time on the more fundamental level.

1. " I disagree that you have identified the core of the big question. Non-numeric representations are nothing new; we have plenty of examples.The challenge is not representation, but /arithmetics/ in the representational space."

Of course, the operations in the representational formalism are important but are supposed to be part and parcel of the formalism itself. However, I suggest you are wrong about "we have plenty of examples", since  non-numeric "representations" you know cannot be called "representational formalisms". Unfortunately, we have been using the term quite frivolously.

2. "most of us have to spend most of our time on less central questions with better prospects of answers in the immediate future."

Good luck!

Personally, throughout my life, I have been most concerned not with the survival but with making real progress in the field. But of course, at the very least, one has to be able to be honest with oneself in separating the two. ;--)

(1)  Did you define representational formalism in the first place?

I can see the limitations of the representations I do know, and corresponding capabilities of numeric sets, but I wonder where you find the definition to exclude them categorically

Hans,

Here is somewhat outdated paper I wrote ten years ago on representational formalisms:

http://www.cs.unb.ca/~goldfarb/ETSbook/ReprFormalisms.pdf

In general, I want to emphasize that a representational formalism is very gradually developed with some applied purposes in mind. So far, we have had basically a single applied representational formalism, the vector space formalism, whose intrinsic structure makes it unworkable for the purposes of classification/PR (because an adequate concept of class cannot be introduced in it). Other "representational" formalisms do not really deserve that name.

Machine learning deals with the construction and study of systems that can learn from data, rather than follow only explicitly programmed instructions where as Pattern recognition is the recognition of patterns and regularities in data. [wiki]

Machine learning, data mining, and pattern recognition are sometimes conflated. [Wiki]

Pattern Recognition has its origins in Engineering, whereas Machine learning grew out of computer science. However, these activities can be viewed as two facets of the same field. [C.M.Bishop's textbook on PR and ML]

Pattern recognition is nearly synonymous with machine learning.[wiki]