I have a time series of recorded hourly temperature of 22 months with few missing cells. These missing values hardly goes beyond 10 hours and out of 16056 data points some 623 are missing. I interpolated the values for single cells but how to do it for consequent 7-8 missing cells.
As the diurnal fluctuation of temperature follows a sinusoidal, kind of, pattern, is there any way to detect this pattern and fill the missing nodes.
If you describe your series with a regression model (using the trigonometric approximation with the sinusoidal pattern you mentioned) then you will be able to find missing points as predictions based on the regression model (after you fit the model).
Dear Deepak Singh Bisht, the sinusoidal diurnal fluctuations doesn't say anything about the nature of hourly temperature changes. I would suggest you at first to form artificial gaps in the continuous data and find a suitable model of the process to minimize the difference, then to use the selected model for missind data. Let us hope, that the model for different times of the year will be unchanged. Good luck!
You can rfer the paper of data imputation. Filling the missing values are called as imputation task. So you can find several paper in google. Auto-associative neural network is very popular approach to resolve these issues.
You can perform the approximation of vector based on the complete set of values throughout required values.
e.g. see approx() function from R language.
For consequently 7-8 missing cells you can take monthly average , or if you have another stations with availbale data then you can do regression analysis as well.
You can interpolate several consecutive cells using a regression model. One that is able to consider a sinusoidal pattern is Gaussian processes with a periodic covariance matrix. Gaussian processes not only give you a point estimate for each missing value but also a variance. Since you might have several points to interpolate from in your series you might want to use sparse Gaussian processes, especially if you are concerned with training time. Even if not, standard Gaussian processes can hardly deal with a tenths thousands of training points. There are many available implementations of Gaussian processes including several flavors of sparse notes. Hope that helps!
Thanks everyone for their suggestions. Is there any harm in filling missing cells using spline interpolation in MATLAB ?
All of the above methods are various ways of interpolating the data into the missing areas. If you have a good physical based model then it's best to use that. If you don't Matlab's cubic spline interpolation (pchip) will usually give visually pleasing results.
The question you need to ask your self is this. Given an interpolated answer is there a way I could disprove it's result, that is, can I show its wrong. If your judgment of the interpolated point is just by eye, Matlab's method is probably fine. If your judgment is based on some model, you should used the model to interpolate.
Either way, always make it clear if a data point was measured or interpolated, as interpolated values are only as accurate as the underlying assumptions.
Deepak,
Regression models have worked well for filling missing water level data, and where you need a more flexible model, neural networks are good. If you're interested in further details, you may want to see this report: http://pubs.usgs.gov/sir/2014/5226/
- Badges
- Science topic