It is important that replicate data be examined for presence of outliers always. Where available, the outlier should be expunged before determining nmber of results (N), Mean (X) and Standard Deviation (SD) etc.
While working on a replicate data recently, I observed that for any set of data (N = 3), provided that the two values/results are exactly the same, the third value/result will by default be an outlier, regardless of how big or small it is from the other two values.
Let's say we have the following replicates from a given analyses/measurements:
A). 3.0, 3.0, 6.0
B) 3.0, 3.0, 1.0
C) 3.0, 3.0, 2.9
Now, in data A, it is obvious that 6.0 is an outlier, and similarly in B, 1.0 is an outlier by mere looking. This can also be confirmed by using Dean & Dixon's Test (Q-Test).
In replicate C, 2.9 may seems not be an outlier due to its closeness in value to the 3.0. In another words, the difference between 3.0 and 2.9 is not significant and thus 2.9 may be considered not an outlier. However, a check on 2.9 using Q-test will prove that 2.9 is an outlier.
Now, my questions are;
Hello Naziru,
Can visual inspection of data detect outliers?
Maybe, if:
1. The data set is of small to modest size (and well organized);
2. Grossly "impossible" values are recorded (Frank Wilcoxon's "outright liars"), such as an adult human stature of 4.52 meters;
3. Data come from a mixture of two or more populations that are very different; and/or
4. One gets lucky.
Can visual inspection falsely flag cases as outliers, or mistakenly fail to identify outliers?
Yes, if:
1. One is unaware of what normal case to case variation is for the process or population under investigation;
2. Measurement errors exist in the capture and recording of the data;
3. The sample variation is wildly larger or smaller than that of the population (e.g., not representative);
4. The sample is too small for dependable estimates of score variation; and/or
5. One is unlucky.
I think your examples indicate three important concerns about the Q-procedure: (a) it takes the observed range of scores as a virtually infallible estimate of variation; (b) it presumes is that the scores come from a normally distributed population; and (c) making inferences about outliers in a sample of 3 cases is very likely a bad idea.
Good luck with your work.
You should not remove a datum just because it is different from others. Please define what you mean by outlier. If you are worried about their impact, consider a different loss function. If you do believe it was produced in a different manner, removing can be okay (e.g., response times when the person sneezes), but be careful.
- Badges
- Science topic
- Similar topics
- Geoscience
- Atmospheric Sciences
- Climate Science
- Climate Change