Dear Lev, I would not completely agree (nor disagree) with your claim. My rationale is that in order to train some models, the notion of a "class" must be incorporated in the training data. Of course, one may then say that the notion of "class" is part of the data, rather than part of the model. But if you removed this part of the data, it would be impossible to train any supervised model (SVM, ANN, Bayesian classifier). Now, if you consider unsupervised methods (KNN, etc), they usually rely on the concept of a "class" much more than supervised methods, since they are expressed in terms of a labeling of the samples or a partitioning of the sample set.

On the other hand, we have regression methods which instead of assigning a discrete label to each data point, they only attempt to approximate an unknown function which depends on the data. In these methods, the concept of "class" is usually irrelevant, or at least not strongly enforced.

So my answer, in conclusion, is that whether the concept of "class" is inherently embedded in a learning method depends on both the model and the application.

Neural Networks, for example, can be used both as classifiers (e.g., with a sigmoidal output neuron) and as regression models (e.g., with a linear output unit).

It's quite an open question, though, and you may get many different answers.

Dear Alfonso, you say "whether the concept of "class" is inherently embedded in a learning method depends on both the model and the application".

This is not at all an acceptable situation in science. Since the concept of class is, in fact, the central concept here, the formalism itself must offer its formal definition, and that definition should also be satisfactory from the applied point of view.

Lev, when I say that whether the notion of "class" is essential to a given method depends on the method itself, and on the application, I do not mean that the concept of class is not clearly and formally defined. For those methods for whom the notion of class is required (i.e., classification methods), have a very precise and formal definition of what a class is.

For example, a class can formally be defined as a subset of the data space, so that the set of all classes form a partition of the data space (i.e., any two classes are disjoint and the union of all classes equals the data space). A class could also be defined as a label value (taken from a finite, discrete set) that is assigned to each data point. Both of these definitions are in fact equivalent. In general, the goal of a classifier method is to find a function from the data space to the set of class labels, so the concept of a "class" is inherent to these methods.

In contrast, in regression methods, the concept of class is often non-existent, so it does not have to be formally defined. Here, the goal is to find a function from the data space to a continuous space (e.g., the real numbers). In this case, a partition of the data space in classes makes little sense because you may end up with as many classes as data points.

Of course, what I am saying is an oversimplification of all the kinds of learning methods one can find, and for some methods, the distinction between classification and regression may not be so clear (again, take neural networks as an example).

Maybe I am not understanding your original questions correctly, but in short, I would say that the concept of class is clearly and formally defined for classification methods, and not required for regression. However, I do not understand how one could design a classification method where the concept of a class is not well defined (or not defined at all).

@Alfonso:

"a class can formally be defined as a subset of the data space, so that the set of all classes form a partition of the data space (i.e., any two classes are disjoint and the union of all classes equals the data space). . . . In general, the goal of a classifier method is to find a function from the data space to the set of class labels …

Maybe I am not understanding your original questions correctly"

=======================================================

Let me clarify my question.

None of the conventional definitions---and you mentioned above some of them---can be considered as adequate. Why?

Having chosen the Euclidean vector space as the *data representation space*, any formal/satisfactory concept of class has to be defined in term of the basic underlying vector space operations (scalar mult. & vector addition). This has been the standard, or required, practice in mathematics starting from the 1950s, as proposed by Bourbaki before the Second WW:

the underlying math. structure of space -----> basic concepts

First, let’s introduce the concept of *class representation* as the chosen formal specification of a class (via the above basic operations).

Then, from the applied point of view, we want to demand one very important property, the *generativity*: every element in the class (*and only them*) has to be specified by the class representation. In other words, class representation is what specifies all class objects and only them.

If we abandon the generativity, as is done in ML & PR, the resulting concept of class representation is not very meaningful---and that is why it has not been introduced so far---since included in it are vectors that typically have no meaning as data objects. In other words, in this case, we specify as part of the class many *meaningless* objects, in which case the concept of class cannot be considered as satisfactory (if we are to take this concept seriously). And my claim is that we must take the concept of class much more seriously than has been practiced, since this is the central concept in learning theory and AI in general.

The above inadequacy is not a limitation of any *concrete* vector space learning model, but the inherent limitation of the vector space itself as the *data representation model*: the vector space formalism cannot support an adequate concept of class.

So the main reason we are discussing this situation is quite unusual in science: the conventional, numeric, form of data representation does not allow for a meaningful concept of class, i.e. we have to abandon the vector space as the data *representation* space. As you know, there has been many proposals for other representational formalisms, e.g. strings, graphs, etc., but for similar reason none of them can be considered as satisfactory.

Again, the minimum requirement for an *adequate* concept of class is the generativity of the class representation, and the main reason why the concept of class representation has not been introduced so far is that it cannot be supported by the vector space formalism.

Of course, the concept of class representation is supposed to ensure a stronger link between the concept of class and the data representation and actually clarifies what a class is.

Finally, these considerations led us to the development of the ETS formalism, an informal introduction to which can be found at

http://www.cs.unb.ca/~goldfarb/BOOK.pdf

Lev,

You might be onto something here but I think that, above all, anyone reading this will find very difficult to understand what you are trying to say because we lack a capable alternative approach to contrast with the current methodology. Having read the draft that you have on your website about this subject, I have to say that you still left me wondering this precise element. In that document, you talk about some "more natural" structures or representations for data but I was not able to understand what and how would you work with that. So, maybe the current methodology is inadequate but it is one we know that it works and scientists resign themselves to live with it until something better comes along.

On the other hand, I have to say that the idea of using more natural representations of objects is not completely alien to the PR&ML communities. People have been working on approaches to work with data in non-Euclidean spaces for some time. For example, there are (kernel) methods to work with graphs of relationships and neural spike trains (I have been involved in the latter) precisely because it was clear that a better representation for the data was needed. That said, one must keep in mind that the most natural representation of some data may be the Euclidean-based form in which we currently work with it...

In addition, note that, as Alfonso pointed out, it may not be always necessary, nor reasonable, to have the concept of class. If I'm trying to predict the location of a physical object, which is a continuous variable, then I would never want to impose the discrete-nature of a class or I'll be unnecessarily distorting the problem.

On a philosophical note, I somewhat of disagree (I think) with your proposition of "generativity" of a class. While I agree that it is important for a class to stand on its own representation, I would argue that real systems use both a class' own representation and the dissimilarity (or lack thereof) to elements of another class. Think how many times you've been told about some idea and you thought, "uhm, it kind of looks like that other idea but with some difference here and there". So, is it the same idea or not? Well it depends on what you are trying to do. Think of it as the "is the half glass full or empty" type of philosophical question...

So, I can understand your frustration. At the same time, try to understand ours when one reads your comments. I hope that my comments may help with some common understanding.

Best regards,

Antonio

@Antonio,

1. Have you read the first two chapters of my developing book? http://www.cs.unb.ca/~goldfarb/BOOK.pdf

2. "In addition, note that, as Alfonso pointed out, it may not be always necessary, nor reasonable, to have the concept of class."

I'm afraid you and Alfonso might be irredeemably spoiled by the current ML / PR formalisms. ;--) A thoughtful scientist seriously engaged in classification *must* be concerned about the underlying concept of class. However, you seem to contradict yourself in this: "While I agree that it is important for a class to stand on its own representation". ;--)

3. "People have been working on approaches to work with data in non-Euclidean spaces for some time."

By the way, do you know who introduced the method of embedding of structural data into pseudo-Euclidean spaces?

4. "In addition, note that, as Alfonso pointed out, it may not be always necessary, nor reasonable, to have the concept of class. If I'm trying to predict the location of a physical object, which is a continuous variable, then I would never want to impose the discrete-nature of a class or I'll be unnecessarily distorting the problem."

When you speak of the "the location of a physical object" you are already relying on the conventional spatial motivated concepts. ETS (and quantum physics) suggests that there might be no "continuous variables".

5. "So, I can understand your frustration. At the same time, try to understand ours when one reads your comments. I hope that my comments may help with some common understanding."

It is not so much "frustration" as *great* disappointment with the unthinking attitude of many workers in the field. However, given the historically unprecedented nature of AI *as science* and that we are living during the unprecedented scientific (and economical and social) transition period, and also the inability of the vast majority of people to understand this reality and cope with its pressures, one has to simply accept that reality. ;--)

Lev,

I did read your developing book, and the above comments express all the questions and thoughts that were running through my head as I read it and, at the same time, tried to understand what you meant and challenge my own assumptions.

In the third paragraph of my previous response, I tried to say that I’m not convinced that we always need classification. Yet this seems to be your assumption(?). when you say “seriously engaged in classification *must* be concerned about the underlying concept of class”, I think that “If I’m not interested in classification for a problem, this does not apply to me.” You seem to suggest that every PR/ML problem *is* a classification problem. Even if this true, does one always need to work at the full resolution? Maybe it won’t be perfect, but good enough is fine for many practical applications.

All in all, my response expresses my ambivalent relationship with theory: I’m very interested to challenge myself and learn new things but, at the end of the day, I still need to use whatever I have at hand *now* to get the job done. History and philosophy are very interesting and great food for fought but a company ultimately cares either it can use the results to get something out of it. Put differently, show me how the concepts you bring forth can help and you’ll have my full attention.

Regards,

Antonio

Antonio,

You can sleep well for now. ;--) ETS is not coming yet for the 'public' consumption; it is for the ambitious young turks in science and engineering.

However, I'm trying to get some investments for developing ETS-based search engine.

Have fun!