My team are now working on our thesis in relation to forecasting particularly in water level. We haven't yet fed our data in our model because there are anomalies found in the dataset. Anomalies found are the time inconsistency and the missing data. We have more than 20% of missing data and we haven't found any articles to support if this is just okay, if we could still get a better result. We hope that you could help us in these issues. As far as we have researched, there are no studies regarding this issue specially in hydrological field.
Do you know with artificial intelligence, geographic information systems and statistical software?
Various articles have been published in this field that I would send them to you if you want to.
If you are in a hurry, send me your data in an Excel file.
Missing data in hydrological studies are not common. What is critical is the purpose of the use of the data without misrepresenting. We have used hydrological data modeling tools to fill the gaps in data. In many cases, with proper statistical tools this can be done. However, this approach depends on the width and depth of the data and how well our understanding goes in the data. For example, we had a 40 year data set where there were holes which we figured out based on other temporal points. This is acceptable statistically. However, if 20% of the data is missing on critical parts within a short (relative) period (1 or 2 years), this could be a problem. We are subject to misrepresenting our data in such cases. Very dangerous to fill those gaps without proper knowledge on the dataset. I know this answer is long and not straight, nevertheless, beware of the fact that missing data should be handled very carefully based on the interpretation and purpose. In some situations, if the gaps are experienced in known trends these could be interpolated, through several statistical packages. Data from similar studies or surrogates could be used to provide evidence.
Anomalies in the readings of water levels can be obtained for several types, most notably the presence of living organisms that may affect the stability of readings
On average, an abnormality rate of 20% is big but somewhat acceptable
I wish you good luck
Well,
In a continuous recording of the rainfall of a station, some data are missing for different periods, in order not to lose the continuity of the information, it is necessary to estimate the missing values. so you can use: Calculation of averages or Linear regression on the measured variables.
For the missing value in each station i get a maximum 10%
Good luck
You have to make the best of the data you have. As long as you are up front about the limitations in your data, it is fair game. There are things you could do, however, to check how robust your result are. As suggested, you could interpolate the data using linear or another technique. I would compare the results with raw and interpolated data (with more than one technique) to check if the results change much. If they don't, then that would give you more confidence. If the do not, then at least you would have an estimate of the uncertainty range. Also, I would not try extrapolating since this could be very risky. All the best.
- Badges
- Science topic
- Similar topics
- Geoscience
- Hydrology