Dear,

How imbalanced your data was? What is your purpose?

You over-sampled the minority class and that may influence the results of Cross-validation. Shall you consider a combination of over-sampling and under-sampling.

Additional comments:

SVM - Usually a strong classifier, therefore dificult to get rid of overfiting specially if and how you removed outliers/noise;

RF - Prunning is a good way to avoid overfitting, as well as minimum leaf size at some extent;

KNN - Since it is a deterministic algorithm, seems to be affected by the over-sampling;

FFNN - You should look at training/test curves and maybe you can control the overfitting with an early training stop. A dropout layer may also improve the generalisation of you algorithm.

You can also consider ensembles, training different models with all samples of minority class and small samples of majority class.

Regards.

how about normalize the features into range of 0..1?

I think you need to rethink of the features you are extracting. It seems that the features do not provide enough information.

Thank You Mr Reza Fuad Rachmadi

I already standardized my data.

Thank you Mr. Turki Haj Mohamad

I shall have a look into my features.

As someone mentioned about normalization. Yes it is necessary to normalize your data. Try to reduce the dimensions as much as you can.

I would like to know what kind of data are you working on?

you did everything with what you have. so you need feature engineering to get more information as Turki Haj Mohamad mentioned

You could also try omitting PCA and see what happens. Additionally you could try looking into NNs. Even using SKlearn MLP should be enough to gauge their performance before moving to Keras or whatever.

One last thing: do you have any metric of how humans would perform using a similar dataset? Otherwise, you can't really tell if 60% is good or bad.

And, of course running clustering WITHOUT specifying 5 classes to see how many clear clusters the data says there is, is worthwhile. In which case again, making a split and then sub-classifying the subset is the way to go.

Thank you everyone.

My updates: PCA is not at all helping. After removing outliers in all features. I am getting lttle good results but as you people mentioned I am doing Feature engineering by extracting each feature and trying to train the algos.

By the way just for Info. I am working on NLP trying to assess readability difficulty level of online News articles. As of now I am using Cambridge English Proficiency tests for different levels containg only 330 articles and the other corpus is story books according to reading levels(contains approx. 1750 articles).

I shall update you guys after doing some feature engineering and Removing outliers(Univariate test)

Thanks alot Guys for your valuable inputs.

Then they are likely not clear classifications as much as bins of a continuous score. Like grades. So could your issue be perhaps a bit like misclassifying a 69% as an A grade and a 70% as a B grade? When actually your predicted SCORE could be quite close to accurate. I'd also consider that the knowledge of how the "TRUE" levels were assigned may be critical if you are wanted to model that. ie. Was it using average difficulty of text or MOST difficult part of text score.

Hello Mr David Mccann,

The true levels are assigned as per average difficulty of the text.

Well actually there is no gold standard corpus extising for this type of assessing problem, so every researcher is using their own available corpus.

Thanks and Regards

Udaysimha Nerella

Dear Udaysimha:

You address the imbalance problem in your data? You say that have 2 classes with more samples that other. Mayor classification algorithms have poor performance in imbalance data.

Here is some info:

• How to handle Imbalanced Classification Problems in machine learning? (https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/)
• Class Imbalance Problem in Data Mining: Review (https://arxiv.org/ftp/arxiv/papers/1305/1305.1707.pdf)
• Imbalanced-learn: a python package offering a number of re-sampling techniques commonly used in datasets showing strong between-class imbalance (https://github.com/scikit-learn-contrib/imbalanced-learn)

Best wishes

Dear Mr. Vladimir Milián Núñez,

Thank you very much for referring nice articles. I shall have a look at them.

Regards

Udaysimha Nerella

Hello, I have meet this problem before.

First, maybe the problem is hard to classify, or your features is not good for the problem.

Second, Sklearn packages is not as good as other software such as Weka. may be you can try some software such as Weka or SPSS Modeler.

Best,

Hi, have you looked at your data distribution? Depending on the distribution some features may become meaninless when Rescaled, i.e. 0-1 normalization. Try Mean normalization instead.

Also, have you tried One-vs-Rest approach? It's a good approach when you have a multiclass classification problem with few classes.

Mr Ricardo has already mentioned about the data distribution problem. If you have a imbalanced data set(Skew dataset), first you need to address this problem. You can apply any method to handle skew datasets.

You can try ATOVIC classifier permitting to valorise features as benefit and cost. The algorithm is defined at: Article Amended fused TOPSIS-VIKOR for classification (ATOVIC) appli...

Thank you all. I am sorry for not replying for your suggestions. I was badly working all possible ways to fix.

I got out of the problem to an extent.

My old dataset can't provide any better accuracy for my selected features. So i changed my dataset and randomly duplicated the values to address the imbalance of classes.

Now i get good accuracies but with over fitting.

1. SVM-poly kernel: training: 94.6 and testing: 86.1

2. RF: 100.0/91.4 training/testing

3. KNN: 100.0/89.3 4. Feed forward deep learning: 98/92

all these are with 5 fold cross validation.

can you assist me how to get out of overfitting

Thanks and Regards

Udaysimha Nerella

Dear,

How imbalanced your data was? What is your purpose?

You over-sampled the minority class and that may influence the results of Cross-validation. Shall you consider a combination of over-sampling and under-sampling.

Additional comments:

SVM - Usually a strong classifier, therefore dificult to get rid of overfiting specially if and how you removed outliers/noise;

RF - Prunning is a good way to avoid overfitting, as well as minimum leaf size at some extent;

KNN - Since it is a deterministic algorithm, seems to be affected by the over-sampling;

FFNN - You should look at training/test curves and maybe you can control the overfitting with an early training stop. A dropout layer may also improve the generalisation of you algorithm.

You can also consider ensembles, training different models with all samples of minority class and small samples of majority class.

Regards.

Hi Mohammed Deeb ,

Did you publish the paper? Can you share me the paper?

BR