Hi Satya,

this is not exactly a k-cross validation technique, however you can use, say 70% of your data to train your model (start with the first time sample and select the first 70%) and predict the next sample (e.g. use data 1900-1950 to predict 1951).  Then slide your window one or more samples forward (depending on the strength and extent of your serial correlations) and use a same-length window to train & predict next sample, e.g. (1905-1955 to predict 1956).  Keep the amount of training data constant and move forward until you hit the limit of your data.  Aggregate your error (MAPE, MSE, take your pick) and this should give you an out-of-bag estimate to start with.

This should be a good place to start.  You can control the amount of training/testing data by adjusting the size of the training window (with 70% for training, you test your method on 30% of your data).

- Martin

Hi Satya,

this is not exactly a k-cross validation technique, however you can use, say 70% of your data to train your model (start with the first time sample and select the first 70%) and predict the next sample (e.g. use data 1900-1950 to predict 1951).  Then slide your window one or more samples forward (depending on the strength and extent of your serial correlations) and use a same-length window to train & predict next sample, e.g. (1905-1955 to predict 1956).  Keep the amount of training data constant and move forward until you hit the limit of your data.  Aggregate your error (MAPE, MSE, take your pick) and this should give you an out-of-bag estimate to start with.

This should be a good place to start.  You can control the amount of training/testing data by adjusting the size of the training window (with 70% for training, you test your method on 30% of your data).

- Martin

I can read these two things: 

1. An answer on stats stackexchange forum: http://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection

2. This paper on cross-validation for model selection, see section 8.3: http://projecteuclid.org/euclid.ssu/1268143839

Hi Satya,What martin has suggested is normally followed for forecasting with a time series data; you cannot mix up the order of the data because we try to find the nature of dependence of the data on the previous sequence of values. You can see the two references given below.. They may be of some help to you.

T. K. Bhattacharya and T. K. Basu,

“Medium range forecasting of power system load using modified Kalman filter and Walsh transform”, Electric Power and Energy Systems”, vol 15, no 2,(April 1993) 109 -115 ISSN: 0142-0615.

S.G.Bhirud and T.K.Basu : “Long Range Forecasting of Hourly Power System Load by Artificial Neural Network via Discrete Cosine TransformIE(I) Journal-EL, Vol. 83, September 2002, pp.70-77

Hi Satya

You can indeed use k-fold cross validation in time series to estimate errors or to build ensembles - but only for autoregressive models. There all relevant information is contained in the lags of the input vector. Please see our paper on Crogging (Cross-Validation Aggregation) at IJCNN and a journal one forthcoming:

Crogging (cross-validation aggregation) for forecasting - A novel algorithm of neural network ensembles on time series subsamples. Devon K. Barrow, Sven F. Crone, IEEE International Joint Conference on Neural Networks, IJCNN, 2013

For moving average processes (i.e. recurrent neural networks, ARIMA models wiht MA(q) terms or Exponential Smoothing), you can follow the approahc by Hyndman in Boosting on the residuals:

Boosting multi-step autoregressive forecasts, Souhaib Ben Taieb and Rob J Hyndman, International Conference on Machine Learning (ICML) 2014, to appear.

Hope this helps, else contact me or Devon directly.

Best wishes

Sven

Really great paper Crone...

For my point of view, your question is very good and very important! Unfortunately, I have read some articles about time-series forecasting (mostly - finance field), in which authors have used cross-validation "directly". Obviously, it isn't correct! It is impossible to put earliest point to the validation set and the more last - to the training. But it seems me, it isn't problem to separate (automatically) all point between training and validation and simultaneously to check, that for ALL points from current combination should be Time(Tr) < Time (Valid). Or, more easy  to select some value T* and - all points with Time < T* are for Training and all point with Time > T* are for Valid. 

Hi Sergey. What you mean is a hold-out cross validation, which is most wideyl used but not clearly separated from k-fold CV.

From our view, hold out CV is in effect similar to 1-fold cross validation (so you do not switch folds to parameterise the model on fold 1 and estmate the errors on fold 2 and then do it vice versa, but parameterise on 1 and estimate on 2). In this case both folds are normally sequentially sampled autoregressive vectors. You could also do a monte carlo cross-validation hold out method where the instances for each fold are randomly selected and not sequentially. 

Hope this helps

Sven

Dear Sven, you are fully right - second from proposed of me approach is hold-out, and usually for time-series we use it. But first approach (to generate all possible combinations of k-fold and to reject non-realistic, s.t. for some point of Training time is more than for some point of Validation) is similar normal k-folt. It is very easy to implement it!

Regards, Sergey.

Hi Satya,

You can refer some of our publication listed in research gate to get answer.

regards

they should order according to their time index