Dear all,
I have a huge database of crude hourly meteorological data, and I am looking for some tutorials for processing it, filling gaps, detect outliers, etc. preferably using R (but I could also use Python and Julia).
Any suggestion?
Thanks in advance!
Hi Paolo,
To fill gaps in your data you can complete with the average value of those variable. This method is valid depending on number of gaps and the variable. Furthermore, you can use boxplot or simmilar methods to determine outliers.
I can help you, providing a basic script in R. Just let me know.
Best wishes.
There is this site I have been learning from. It has a chapter in filtering outliners and missing data; how missing data should be treated when they affect the outcome and so on.
The whole book can be found on.
https://r4ds.had.co.nz/
And the chapter with outliners and missing data are in:
https://r4ds.had.co.nz/exploratory-data-analysis.html
Python provides with high-level functionality to meet all your needs, see e.g.: https://github.com/simongeek/PandasDA
http://meteo.mcgill.ca/~cmccray/python.html
Anaconda Python distribution makes everything easier (https://www.continuum.io/downloads).
Best
I have specific code that i can share. I normally have ended processing the data with my own code. For single missing observations i interpolate using the two observations around it, for other things i have tried backcasting/forecasting using SARIMA / State space models.
Thank you Luis Fernando Chaves , these are not single missing obs. Any help will be appreciated. Best
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R