The effectiveness of any dimensionality reduction technique depends on the characteristics of the data set. PCA, factorizations, transform methods, etc. will exhibit different performance. Dimension reduction is in a sense, a form of data compression. You should see if your data possesses any symmetries that might be exploited or if certain basis sets can be used to represent your data, where most coefficients would be zero.

You mentioned that PCA did not improve accuracy, but did you achieve similar accuracy with only a few basis vectors? if so, then you have improved your situation, no?

Hi George,

Thanks for the explaination.

I used PCA for credit card data and I extracted 15 PCS (Originally there were 23 variables) which was explaining 90% of the variability, however, i didn't consider eigen values above 1 which were only PCS and they were only 60% of the variability.

I got 80% accuracy for KNN model with PCA and without PCA.

Please suggest dimension reduction techniques for SVM.

Thanks,

VB

Kernel tricks?

Nowadays, if you have enough data, the best way to dimension reduction is autoencoder. If you are using images, try some variety of convolutional autoencoders.

Thanks Eanes. Can I use autoencoder for non-images, if so what are those.

Penalised SVM and t-sne will work for SVM model.

Yes, Vijaya Ba, you can. Broadly speaking, for generic data you can use deep neural networks with decreasing/increasing number of neurons in hidden layers.

See this papper: Article Auto-Encoder Based Dimensionality Reduction

and this link: https://blog.keras.io/building-autoencoders-in-keras.html

Sure thanks Eanes. Based on my understanding SVM can be used when more number predictors than target variable.

So, I can try penalised SVM and t-sne, specifically for SVM , if I want to predict fopr plant growth.

Can auto-encoder can be used in case of SVM specifically or for in general for all models.

Hey there,

Previous works have demonstrated that Partial Least Squares is able to achieve better results than PCA (mainly when using SVM). For instance:

@Vijaya Ba

The dimension reduction techniques do not make miracle with your data. You should not expect that a data reduction method will increase your accuracy. These methods are used to decrease training time and data complexity. There can be circumstances when a classifier may perform better after data reduction, but it is not a free lunch for every case.

And honestly I don't see that reducing 23 variables to 15 will do anything better. It does not reduce the dimensions too much.

Hey Vijaya Ba,.

Best of my knowledge, SVM is a recipe which helps you to train the data and not for dimensionality reduction; an appropriate kernel function can be used in the SVM, which helps to reduce the dimensionality.

In your case, reducing the 23 variables to 15, you can use “Facotr Analysis”, which is the advanced technique compared to PCA. Alternatively, you also can use “Regression (Enter) Method”, which removes the variables that are not play a significantly role in your study.

Simply you can use these both methods or you can employ these methods through SVM recipe. If you use directly (without SVM) you can avoid the computational complexity.

Hey Vijaya,.

Best of my knowledge, SVM is a recipe which helps you to train the data and not for dimensionality reduction; an appropriate kernel function can be used in the SVM, which helps to reduce the dimensionality.

In your case, reducing the 23 variables to 15, you can use “Facotr Analysis”, which is the advanced technique compared to PCA. Alternatively, you also can use “Regression (Enter) Method”, which removes the variables that are not play a significantly role in your study.

Simply you can use these both methods or you can employ these methods through SVM recipe. If you use directly (without SVM) you can avoid the computational complexity.

Sure, thanks all. I will try experimening all the mentioned techniques to check the accuracy and as it's performance is based on the dataset I am using, different trial and error techniques needs to be performed.

Many thanks for valuable suggestions.

Regards,

Vijaya

Dear Vijaya Ba,

Please post your results when you have them. It's interesting to know the comparison.

Thanks,

Son

you will find some technicals method in this thesis : http://essay.utwente.nl/84660/1/kolluru.pdf

good luck