There is some invalid data in the orginal data series. For example, the 1 minute series is like 1 2 3 nan 4 5 ....Then if i wanna get the 5 minute average, should i just drop the nans and mean the rest?
How you treat the missing data depends upon time series and the importance of the interval length for the average. A rapidly changing time series may be greatly effected by the choice of replacement of NaN. One method of replacement is to use the mean of the other four minutes. You can test the effect by running averages across the NaN location. Running averages may be a better solution than simply averaging each five minutes.
I checked the data from NGDC and found that there were two cases they delt with. One is just to drop the invalid data and mean the rest. The other is to drop all the values in the interval whenever there is a valid data point. I dont know which is better. Anyway, thanks for your time and answer.
The answer to the best substitution or ignoring the Nan remains with the effect on the data series and your objective. You may find answers in the literature on image enhancement.
- Badges
- Science topic