Hello !
In a work, I have to benchmark different algorithms to fill in missing values in time series.
I insist on the fact that this is imputation and not forecasting.
In my case, I have access to 15 years of complete temperature data from 20 stations.
I have different algorithms that, knowing the position of missing values, try to complete the missing data.
However, these algorithms have some parameters to be set. So I want to implement a classical method such as k-cross fold validation.
Moreover, these algorithms have to be tested in a classical case of time series completion. Indeed, here is a typical drawing of time series data completion:
http://prntscr.com/106qpc2
Each line is a time series. (note that some series are and must be complete for the benchmark)
In red are the missing data to be completed, and in green are the unknown data. In practice, these algorithms use the notion of space-time to complete missing data.
However I am facing a problem. Indeed, the method of k-cross fold validation takes randomly data and it is this purely random character which poses me a problem. Indeed, as shown on the drawing, some algorithms could work better if I take my data completely randomly. Whereas in reality, they will only be tested on a template as on the image. Indeed, I know in advance that some of my algorithms work very well when blocks of several years of known data are present and don't work at all when purely randomly selecting the known data.
Do you have an idea of how to parameterize my algorithms to be optimal on a data framework such as the one on the image?
Thanks in advance
(Feel free to ask me questions if I haven't been clear enough)
You can use the concept of relative performance applied to time series.
One approach is to use the AvgRel-metrics proposed in Chapter 2 of (Davydenko, 2012), see pages 63-64 in particular:
Thesis Integration of judgmental and statistical approaches for dem...
The AvgRel-metrics assume you use the geometric mean to average relative performances and use the loss function you used to optimize forecasts in order to evaluate forecasts. You can use the AvgRelMAE or the AvgRelMSE.
So the AvgRelMAE or the AvgRelMSE compare the performance of a method against the performance of the benchmark using a reasonably well-designed approach.
In your case the questions are:
1) How to calculate the MAE/MSE for the missing data for the benchmark?
2) how to calculate the MAE/MSE for the missing data for the method being evaluated against the benchmark?
Here you just need to simulate: using known values try making it missng and then impute and compare with real outcomes. The benchmark may be the mean of available cases.
Moreover, the AvgRelMAE/MSE will depend on time series features and the number of values missing. But the general approach I recommend is to use the AvgRel-metrics defined in (Davydenko, 2012). More on AvgRel-metrics here (see relevant slides):
Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...
https://datascience.stackexchange.com/questions/89827/time-series-imputation-benchmark
Sudhir Yadav Why send this link? Yes I asked the question on another forum
Andrey Davydenko
Thank you for your response. However I would have liked more to have an idea to start separating my train/test set.
Is it correct and possible to split my set in 2 :
One part over 10 years and another one over 5 years.
Then simulate a lot of times (K times) missing data like the template (screen) on the 10 years data. Select the best parameters for each algorithm, then simulate once the missing data on the 5 years data and choose the best algorithm.
Is it conceivable to do something this way?
What type of missing data do you have? Is it missing-at random (MAR)? https://en.wikipedia.org/wiki/Missing_data#Missing_at_random
Andrey Davydenko
The type of missing data that i have is like on the graphics that i made : http://prntscr.com/106qpc2
Values are missing per block. A block of values can be missing at the beginning of a time series, at the end of a time series or even in the middle. There could also be a missing block at the beginning and at the end of a time series.
The most important thing is that they are blocks of missing values and not isolated values.
I'm not sure if it fits exactly the definition of MAR or MNAR?
Please see the wikipedia definition. If the variables are missing at random, it's easier to simulate the missing values and then compare modeled values with real outcomes.
- Badges
- Science topic
- Similar topics
- Statistics
- Time Series