I'm using SVM and (neural network) for a time series prediction data-set in MATLAB R2016a with 800 samples. Currently I'm using 10-fold cross validation and grid search to find best SVM parameters. I'm using 90 samples (after this 800 samples) as out-of-sample to check performance of final model using best SVM (and neural network) parameters and training my model on whole first 800 samples.
The test accuracy of final model (10-fold cross validation) is about 98% (sensitivity and specificity of about 98%) but when I check designed model on last 90 out-of-sample data (which trained using whole first 800 samples) I have a poor accuracy (about 55~59% total accuracy, sensitivity and specificity). This is daily forecasting of a financial market. Why I have this behavior? I checked normal k-fold cross validation and sliding window validation . I had mentioned behavior (poor out-of-sample accuracies) in two methods.
Hi Eghbal, have you tried checking different kernel functions? Do you know if some sort of intervention happened in your testing data? In this case you could try a variation in the code in order to work with an on-line approach for the SVM regression.
.
maybe some lack of stationarity ?
since your out-of-sample data comes after the training sample if i understand correctly
.
oh, and cv is not a silver bullet against overfitting ... see
http://ai.stanford.edu/~ang/papers/cv-final.pdf
.
you could check if your setting might overfit the cross-validation (too many parameters being searched through for too little training samples) by interleaving your out-of-sample data with your training sample, hence alleviating the possible lack of stationarity
if you get consistent results this way, i would then suspect lack of stationarity on the data
if your results are still inconsistent, i would reduce the search space or go for nested cv
.
For non stationary time series, you can apply different methods of transformatiin such as Relative Difference in Percentages (RDP), Z-score and Natural Logarithm transformations befire implementing the SVR algorithm.
Please see the attached files which deal with application of SVR to non stationary financial time series.
For some unbalanced data sets, accuracy may not be a good criterion for evaluating a model during cross validation. Look at the link below to get some ideas:
https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/eval/index.html
- Badges
- Science topic