My dataset consists of EEG electrode power features in all power bands(alpha, beta, delta..both relative and absolute) and source power features (obtained after sLoreta analysis) in addition to connectivity strengths between the different sources (brain regions). There are as many as 20k features in all.
If i have to predict disease (dementia) based on all above features, what approach will yield best accuracy on test sets? I initially thought that maybe i must fit seperate classifiers for each type of feature set (after dimension reduction) and then use the output probabilities obtained to write a meta classifier on top to predict the final disease state.
However, i think that may perhaps not be so great as all features are correlated (as source estimates and connectivity measures are obtained from the electrodes themselves). Is this correct?
I used KernelPCA to select a few components from the entire dataset and then run a classifier on top of the transformed dataset with cross validation. I get an accuracy of around 75% only on test sets. I have to improve accuracy atleast by another 15%. I used extremely randomized trees but the accuracy was not that much.
What other approaches can i use?
I am looking for a good discussion on possible approaches and/or a sample solution. Thank you.
To understand your question correctly. Are you looking for ways to pre-process your data?
@Roman Liessner no, the eeg time series is already cleaned and pre processed. FFT transforms are applied to extract frequency band level power features at the scalp level. Source estimation is done to find source powers at 68 ROIs in the brain surface. Connectivity analysis is performed to extract connectivity strength measures between the brain ROIs in the different frequency bands. So, in all, there are 20k features altogether.
I have so far used dimensionality reduction techniques like KernelPCA, PLS, ExtremeTreeClassifier to reduce the feature set size. KernelPCA with linear kernel (akin to PCA) and logistic classifier gives about 86% accuracy on test set. But, almost all other classifiers applied on the reduced feature set using various other dimensionality reduction techniques perform rather poorly.
Okay, which classifiers have you tried so far?
Deep Learning? Xgboost?
thank you. I tried PLS (as classifier), KernelPCA with logistic classifier(by far, the best accuracy using linear kernel, rbf does not perform well), ExtraTreesClassifier in sklearn. I have not used any deep learning so far. I will try the Xgboost.
PLS and ExtraTrees perform very poorly (roc_auc_score
Yes, try these two out. With keras you can build deep learning models very fast.
In addition you can try to find even better features.
A hyperparameter optimization could improve your classifier. However, I would try this last. First find good features and a suitable model.
All the best for you.
thank you Roman Liessner , i tried Xgboost. I learnt xgboost, thanks for the recommendation and understand that it's a powerful tool. But in my case, with around 20k features and around 200 samples, the classification auc score is less than 50% on test data. it trains well with an average auc of atleast 85% but test performance is poor. i read that such advanced models need not be used unless the data is somewhere in a few GBs. My data is around a 100 mb at max. Also, i read that it is prone to overfitting and perhaps that maybe a reason for the low test auc score. I did try to do the hyperparameter tuning for 10-15 parameters in Xgboost but the classifier gets stuck somewhere around the 67% mark on the test data at best. I really do not know which other features in xgboost to tune to get the classifier upwards from 67%.
I will try the keras with a deep learning model. But, my question is, given a CPU with lot of RAM and 8 cores, is it really this hard to fit a good model with around 20k features. From what i read, xgboost and deep learning models are fit on data with a million features or more. So, perhaps, i think this is indicative of my not so good skills in ML perhaps :) where i can use simpler models (like a logistic lasso which seems to perform better with a 79% accuracy on test!)
if i had the time, i would try to do an anova analysis or factor analysis (between the frequency bands, disease/normal, brain ROIs) but again, factor analysis seems to compute a correlation matrix which breaks my jupyter notebook!
- Badges
- Science topic
- Similar topics
- Cognitive Science
- Thinking