I'm using MATLAB R2016a for binary classification (time series prediction) of a financial case. I have a good total accuracy (70~75%) but specificity is about 90% and sensitivity is about 60% and vice versa. Currently I'm using grid search to optimize my classification model (SVM, neural network, etc.) based on total accuracy. My data-set has balanced samples of binary output.
How can I improve results in this case? Can I use any other performance metric to take into account unbalanced sensitivity and specificity (as mentioned, in some cases I have higher specificity than sensitivity and in some cases vice versa.)?
So, what you seem to want to optimize is F-score. Just use F-score in grid search optimization.
Another option may be to calibrate the probabilities by plotting graphs similar the ones described here to http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#example-calibration-plot-calibration-curve-py
I do not fully understand what you mean by "and vice versa". For a given dataset there is only one specificity and one sensitivity. But perhaps you have many datasets to deal with.
In any case, what you probably want is to give your two different classes different weights, depending on sensitivity and specificity. In R there is the nice classification model Random Forest (https://cran.r-project.org/web/packages/randomForest), which offers the parameters classwt and cutoff for exactly doing this. By the way, Random Forests often give better results than SVM or NN.
If it becomes too tedious to do grid search for all the algorithm and parameter choices, you might consider to use a tuning method or algorithm selection method to find the best algo setting. The R-package TDMR (Tuned Data Mining in R, https://cran.r-project.org/web/packages/TDMR/index.html) is a good possible choice for algorithm selection in data mining tasks.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining