I have climate data such as rainfall and another data such as aquifer volume.
I want to assess whether all of this data is credible without any errors such as systematic error.
What statistical tests shall I conduct ?
if statistical tests show that data has error, does it mean data must have error ?
The evaluation of the accuracy of the data is a complex subject. So many errors are in a database. Without a doubt. Just thinking about the mistakes of the field, the data entry. In general, errors are many and varied.
How to proceed with the evaluation? it is necessary to return to the source from which the data comes. to trace the setting up, Which technique had been used? how was the data collected? How long did it take? .... which is not easy at all. On the other hand, there are a few measures namely the mastery of outliers, missing values, the sample size, the opinion of an expert in the field for his opinion on the reliability of the data.
About the only way to assess accuracy is to do the experiment over several times. If you do it over once and get a different answer then you do not know which experiment has errors, maybe both do. However, some errors can be captured.
Errors come in many flavors.
1) Experimental design errors. Things like selecting the wrong experimental material, or bad randomization.
2) Data collection errors. Things like writing down the wrong values, or improper use of equipment.
3) Data entry errors.
There are many ways to catch these. Look for repeated values, check sample sizes (I ran an experiment with 4 replicates but this replicate has five values, or this one only has 3 because the value of 40 was entered as 4o). Look at outliers. Look for consistency, by this I mean that there is a well documented paradigm Y that everyone has accepted for the last few decades. While it was not part of the research question, does my data support the paradigm?
Know your program. I could type in control for all the control treatments. I could also shorten it to con, or Con, or coN, or CoN, or cOn. A program that ignores capitalization will treat these as the same. Likewise Jan_3 is not Jan 3 is not Jan-3, and so forth.
Pay attention to the output when the program reads the data and the type of data. Some programs will not like a variable name like copy# for copy_number. When data are read in make sure that the data type is correct. Data that is numeric might be character if the program expects a blank but gets a period or NA, or -, or something else. Also if 4o was entered somewhere then a numeric variable might be typed as character.
If the data comes from a process, make sure that none of the rules for that process are violated. If you swallow the orange juice before getting the juice out of the orange then there is a problem in the data.
Look for patterns that should not exist. So in air temperature measures I found values that indicated very cold temperatures in summer. This was usually followed by a note that the sensor was replaced. Now one needs to go through the data and figure out where the sensor started to fail thereby producing bad data. If possible look at output from multiple sensors. So sensors A, B, and D show increasing temperatures, but sensor C shows falling temperatures. This can be also correlated with other events. The water table was rising in this week but no weather station reported any rain. However, at some point you will run out of time for this sort of activity.
Be careful in defining error. Some people will run an experiment and then remove all values that are more than 2 standard deviations from the mean because these are errors. They have a point that the farm crews are not as precise or as consistent as a research scientist might like. However, one should expect unusual values if you look at enough data. In this case you cannot tell for certain if the value is real or a mistake.
Hi, its difficult with rainfall data, since its a sporadic event and not normally distributed over time (other than T, rF etc.). So you have a lot of "true outliers" and your mean and median are differing a lot. For assessing the credability of the data, try to check how and under what circumstances it was collected. Also try to compare your data with other relevant data sources!
Good luck and best regards,
Andreas
- Badges
- Science topic
- Similar topics
- Geoscience
- Asia
- India