Are there any files containing publicly available large volume time series in format of pairs (timestamp, real value) that are suitable for processing in order to detect outliers? I would be interested in real life measured data from sensors, for example hydrology based data such as Conductvity, pH, Turbidity, etc (but not only limited to hidrology, other science fields are ok). The larger the volume, the better.
I suspect that these types of datasets are usually carefully processed to remove outliers before being published on the net. The DTC dataset from high frequency monitoring is a case in point. Much effort goes into processing this dataset. The clean version should be available from http://www.edendtc.org.uk/, however you would probably have to contact the project team to get hold of the raw data with outliers included....
Our solar and meteorological data set is available on line at http://solardata.uoregon.edu/SelectArchival.html. The best dataset is the Eugene dataset. It is a one minute data set contains, solar irradiance, IR, Relative humidity, temperature, wind speed, and pressure. An explanation of the file structure is at http://solardata.uoregon.edu/ArchivalFiles.html.
Thank you very much for both answers. Regarding the solar data set from Univ. of Oregon - I think this match best what I am searching for. However, I have a couple of questions
1) For processed data (quality flag 12) what algorithms/methods were applied on the raw data (flag 11) to become "processed"?
2) What is the reason for keeping in data files both processed and raw data as well? Was not possible to process all data?
3) Is it any filter that can be applied / or method to get the only files with only raw data?
The irradiance data have thermal offsets. We took the thermal offsets at night and assumed there were the same during the day. In reality, the thermal offsets during the day are usually twice that during the night. Thermal offsets give small negative irradiance readings. These adjustments were made to sites with thermopile radiometers. For these data, we flagged the values at 12 or processed. This was not necessary for other meteorological parameters and for sites without thermopile radiometers.
The irradiance data in the processed data are more accurate than the raw data files, but we have those data files if there you would find them useful.
We are in the process of updating our irradiance data calibrations from 2006 through 2015. We hope to have this done by the end of the year and documentation on what we have done, similar to the documentation of what we did for most instruments through 2006.
- Badges
- Science topic
- Similar topics
- Statistics
- Time Series