See the "Curse of dimensionality" phenomenon.

Thanks for your respond how about GMM ? if you look at the example more feature less probability ?!!?

.

a small vocabulary problem here (maybe) :

in your example, the number of "features" (that is the dimensionality of the representation space of your data) is constant and equal to one for the three test sets. What is changed is the number of "instances" of the test sets, that is the number of elements ("cardinality") of the test sets.

now, i am not familiar with your matlab code but i assume that sum(lp) is the loglikelihood, that is the log probability of observing your test set given the model you buit on your training set.

Indeed, under the independence hypothesis for your observations in the test set, the probability is just the product of the probabilities of observing each sample (therefore the sum(lp) stuff) ; since a probability is always less than one, the more elements you have in your test set, the smaller the product of probabilities is.

As simple as that, assuming i understood your question correctly !

.

The feature set is basically what defines ur classification results. From the question u asked, one dimension is that by increasing number and types of features in your feature data set, you may not always have an increase in performance. As an example, there may be a case where the colour of skin has no effect on identification of the object, so this feature is useless at those places and will merely decrease the performance. In case like we have a scenario with different objects like car tyres, boxes for packing etc.. At this place adding the feature, object shape, will be effective along with other features as it clearly describes the object.

@Clerot Fabrice Thanks for your respond. You are exactly true but let me pose my question in different way. Suppose that you have two dimension features matrix with many samples and you would like to know what is the optimum number of samples which are needed. In my case I have 12*20 feature matrix and I try to decrease it every time and see the performance for example I have test it with 10*20, 6*20 and 4*20 to see the performance. What do you think about my method? is it wrong ?

I will really appreciate, if you could help me.

Dear @Nida Rasheed

I have different signal with different frame. I want to know which signal is more important than others that's why I want to study the effect of each signal on the performance. What is your suggestion ?

Thanks.

The signal effect can be checked in many ways. There are different criterion like checking if your data is normally distributed or not, depending on ur type of signals. The effect can be checked individually which is very tedious. The signals can be divided into multiple ranges and you can see the effect of a certain group of values.

Increasing the number of features , doesn't guarantee better algorithm performance.

For example in a high dimensional data, is often represented by sparse feature vectors, with some features carrying maximum information , while others may be correlated or uninformative , thus increasing the number of features is not exactly contributing to better classification or clustering . There are many ways in order to select the best features , have a look at this:

http://www.ijitee.org/attachments/File/v2i2/B0352012213.pdf

There are other ways also which use variance and correlation.

Also you can save a lot of training and testing time by selecting the features that would actually contribute maximum information.

.

"In my case I have 12*20 feature matrix and I try to decrease it every time and see the performance for example I have test it with 10*20, 6*20 and 4*20 to see the performance. What do you think about my method? is it wrong ? "

Sorry, i do not understand what you mean since i do not understand what a "feature matrix" is. Something like the pixel matrix of an image (then, to me, there are as many "features" as there are pixel (assuming one descriptor per pixel) and as many instances as there are images)? Something else, maybe related to subsampling of a time-series ?

Is it a supervised classfication problem where you search for the best representation space ?

.

@Nida Rasheed suppose that I extract feature from each signal however, the number of feature's samples are different between them. For example for signal1 the 10*20 feature's sample and for signal2 5*20 feature's sample. Then, can I say if the standard deviation and mean of feature samples is less, it works better ?

I don't know my signal distribution. I just can say the feature distribution could be modeled by Gaussian distribution (GMM).

Dear @Eric Nunes thanks for your attention. My problem is how can I understand that which feature sets are most informative? Please regard the number of feature samples are different between testing set. Lets say x1=[1 100 1000] and X2=[500 600 700 1000 101] are two feature vector. Based on train=[1 2 1 2 1 2 100 101 102 99 100 101 1000 1001 999 1003]; we know that x1 is better than X2 because It contains more informative feature in training set. But is it fair for comparison ?

Dear @Clerot Fabrice I extracted 20 frequencies from every 10 frames of one signal 10*20 feature matrix.

I want to know which frames are most important that 's why every time I choose special frame set and put it in GMM for find the probability. However, you said " since a probability is always less than one, the more elements you have in your test set, the smaller the product of probabilities is". for this reason, I am confused because I don't know if I select 5 frames with 20 frequency is works better than 7 frames with 20 frequencies ? I am looking for a way to compare it fairly.

.

ok, so you have many signals ("instances") and each signal may be represented by N out of ten possible vectors in dimension 20, each vector corresponding to a frame and each dimension corresponding to a frequency.

now you want to select the "best" representation ... what do you mean by "best" ? what does "works better" mean ? The representation which allows the best performance in a supervised classification task, the representation which gives the "best fit" to the data in an unsupervised analysis ?

note that the likelihood is a product of _instance_ probabilities : the more _instances_ you have, the smaller the likelihood ; this has _nothing_ to do with the number of features ; what i wrote above is that, for a given and fixed number of features (the number of features you use in your training set to build the model), the likelihood is decreasing with more instances (under the independance hypothesis of the instances).

maybe you are trying to make some kind of clustering of your frames ? This might explain why you talk about "features" when i expect "instances" !

.

Let me explain you in other way. Suppose you have a video and you have to extract features for each frame. The features will be mean, min, max etc. For every frame, or signal in ur case, same features will b extracted and depending on the features you have to identify that what is the classification of data. The size or dimension of the feature set remain constant throughout the procedure. The only thing that changes is the input to that feature, i.e signal in ur case. Hope this helps

.

@Nida

then it would be a supervised classification problem where the instances to be classified are videos.

there are different possible representations of those videos but there is a possibility of defining a fixed test set so as to compare the classification performance of different representations

this sounds too simple ... so there must be something i'm missing

maybe the task is a ansupervised classification ("clustering") ?

.

Dear @Clerot Fabrice. I mean exactly the number of feature instance that you mention. suppose the feature is frequency. I want to know how many frequencies are enough for more probability ?

For example we train our GMM with 50 frequencies of 10 signals. For testing our system, 10, 20 and 100 frequency of a signal is extracted and put it in GMM for evaluation. However,100 frequencies have less probability than 10 frequencies !?!. Moreover, we know if we want to compare various signal in term of frequency, 100 frequencies give us more information and discriminative . Then why it has less probability ?

This matter is confuse me for testing my system !!!

Dear @ Nida Rasheed the dimension of feature set is changed because I want to compare [2*20] [4*20] [10*20] probability but the problem is that [2*20] always give me more probability than [10*20]. I am looking for a way to compare it fairly.

.

would this make sense to you (re-using your example) ?

test=[1 2 1 2 1 2 100 101 102 99 100 101 1000 1001 999 1003];

train1=[1 1 1 2 2 2 100 100 100 101 1000 1000 1000];

train2=[1 1 2 2 100 99 1000 999];

train3=[1 100 1000];

No_of_Iterations=10;

No_of_Clusters=3;

[mm1,vv1,ww1]=gaussmix(train1,[],No_of_Iterations,No_of_Clusters);

[lp,rp,kh,kp]=gaussmixp(test,mm1,vv1,ww1);

sum(lp)

[mm2,vv2,ww2]=gaussmix(train2,[],No_of_Iterations,No_of_Clusters);

[lp,rp,kh,kp]=gaussmixp(test,mm2,vv2,ww2);

sum(lp)

[mm3,vv3,ww3]=gaussmix(train3,[],No_of_Iterations,No_of_Clusters);

[lp,rp,kh,kp]=gaussmixp(test,mm3,vv3,ww3);

sum(lp)

that is building 3 models (one for each set of "features") and testing them the same test set (therefore, comparing the likelihoods makes sense).

to me this has nothing to do with "features" but just experimenting how more "instances" contribute to the quality of the model ...

.

Thanks Clerot Fabrice. Then you mean there is no way to normalize the results of different test vectors with different sample size with same training set?

How can I measure separability between different number of instance of one feature vector ?

Besides the "curse of dimensionality", it can also be the case that you are introducing features that are irrelevant to the underlying class label. This may also affect the clustering. See the attached webpage for an explanation and a figure.

http://www.comp.hkbu.edu.hk/~ymc/project/Feature_Selection/project.htm