I am relatively new to machine learning. I am part of an in-class competition on kaggle where the task is to predict churn. I trained my model (random forests 50-200 trees) on the test data and got up to 80% accuracy using a 70-30 split. However, when I predict on the unseen data, I get zero accuracy on kaggle.
For some reason, I decided to invert the labels thinking something went wrong and then I get 100% on kaggle. Can someone explain to me why my model wrongly predicts every case in the unseen data. Confidence level of each prediction is mostly >0.7 (ergo it's not a coin toss).
Dear Olabode,
Please let us understand the data first. Additionally, you can check also whether any previous works done on it or not. Inverting the labels does not seem to be a good idea. Can you please logically justify the reason behind it? Though you have got an excellent accuracy, I would like to finally suggest any mathematical background to be further searched for your works.
Please go in-depth.
Hi Olabode,
Without seeing the data any advice would be not quite accurate. Anyway, my advice is to use same data structure (that you mainly built the predictive model from) when testing the model. Moreover, make sure that your data are well-prepared as this might be the reason behind generating unstable models. Finally, don't rely on the accuracy alone to evaluate your algorithm, so consider, for example, ROC, Kappa statistic, and confusion matrix.
HTH.
Samer Sarsam, PhD.
Hi Olabode,
Since your cross validation is working fine, try looking into the following aspects.
1. perform a 10 fold cross validation, instead of random 70-30 split and do a ROC curve analysis. If your Area under the curve is towards 1.0, the model is strong.
2. Check in the test data, whether the features and their order are same or not.
3. Get rid of outliers, and try making the Random forest model more robust by increasing the number of trees. Also, you can scale the data across samples.
I hope this helps.
Dear Olabode,
Please let us understand the data first. Additionally, you can check also whether any previous works done on it or not. Inverting the labels does not seem to be a good idea. Can you please logically justify the reason behind it? Though you have got an excellent accuracy, I would like to finally suggest any mathematical background to be further searched for your works.
Please go in-depth.
Thanks to all for the responses. As I stated in my question, I am new to the field of data science and Machine Learning. At the moment, I mostly have a high level understanding of ML concepts.
The data is a telecoms dataset with about 15 features and 1 target. The task is to predict customer churn. Dataset is balanced( 50-50). Train set contains 1600 rows, test set is 400 rows long. The distribution of each feature is similar across training and test sets. Used Rapidminer.
Inverting the label was only out of curiosity.
- Badges
- Science topic