When discussing data collection, outliers inevitably come up. What is an outlier exactly? As a definition, It’s a data point that is significantly different from other data points in a data set. While this definition might seem straightforward, determining what is or isn’t an outlier is actually pretty subjective, depending on the study and the breadth of information being collected. But I have a question that when should I delete outliers from a data set?
You can consider the standard deviation. If a certain value falls outside of 3 standard deviations you can say it an outlier. You can delete it or can replace it by the median based on your application.
I don't agree with Morshad Ahmed 's answe. I don't think that this is a good advice.
There is no fixed, objective definition of an "outlier". The definition is just that it is a point "far away from the rest". This is simply a description, and the fact that a point happed to be "outlying" may or may not indicate a problem. There are circumstances in which you would expect outliers, and it would be strange not to see any such outlier.
You can ideally identify outliers from plots of the data or from residual diagnostic plots. Particularily in multivariate data, apoint may not bean outlier in any of the variables, but still could be a relevant multivariable outlier (something you would see only in residual diagnostics).
Having identified outliers allows you checking their plausibility. If these are extremely implausible or even impossible, its better to just excude them (and report that!). If they are not extremely implausible, it might be possibleto check if there could be someother problem with the measurement or recording that can explain a "strange" value. If such a reason can be identified, the outlier should also be removed (report!). If there is nothing obvousely wrong, then the value has been recoreded with the same care as all other values and should therefore be taken as seriously as all other values.
Outliers should actually not be a problem. They should be rare. If they are not rare, then you either have a suboptimal understanding of what an "outlier" is or your assay/experiment/method is really bad and casts doubts on the reliability of your entire study! If you regularily have outliers, and if they are not really rare, they are part of the story. It can be that you have some "contamination" of a different distribution (example: you count the number eggs in turtle nests - there are usually several hundreds. However, some nests are almost empty, with onlyzero toafew tens ofeggs remaining, possibly because they were found by a nest predator. The low counts are outliers. It might be relevant to include the information of the number of "almost empty nests" in the study instead of just removing these observations).
Now, if outliers are rare, then they are not a problem for the analysis. If you have only very little data and you think that there is one value outlying (but not extremelyimplausible/impossible and without having a hint that this is a wrong recording or similar), this might be the only value trying to tell you that the few other values are too close together. In such cases (small samples) an outlier does have a considerable impact, but there is no reason to believe that this impact is "bad", leading to"wrong" conclusions. It could be the opposite! Removing such values can make things worse. This is particularily the case when the underlying distribution is skewed. In this case, there is a higher probability to obseving outliers in the direction of the skew, and removing such values causes bias.
An outlier is an observation that lies outside the overall pattern of a distribution [implying, significantly far from other data points, lower or higher] (Moore and McCabe 1999). Outliers affect the mean value of the data but have little effect on the median, you wish to replace it with, other options exist.
It's not always good to drop a outlier when it does not change the results but when it does affect assumptions, you may drop the outlier. Except you have a prior knowledge about the process and the measurements or suspect sample error(may be on in a count), with a good sample size, removing or replacing the one or two outlier with the median is ok to me!
However, removing often, or when the sample is small and the underlying distribution is skewed could be a big mistakes. A commonly used rule says that a data point is an outlier if it is more than 1.5*{IQR} points above the third quartile[Q3] or below the first quartile[Q1]. IQR is the interquartile range. Am not sure of the justification!
There are several techniques to detect outliers; however, I would like to endorse the Jochen Wilhelm 's answer, especially "If you regularily have outliers, and if they are not really rare, they are part of the story".
Outliers can "speak" to the researcher and they can "say" more regarding our hypothesis .
Seyyed Amir Yasin Ahmadi ,
censoring requires an information about the minimally / maximally possible value you would trust. If the values are from an assay with known dynamic range, it may be applied (e.g. left-censoring for values below LOQ - but these would aczually not be recognized as outliers!).
Robust models are fine if you have a measurement process that is known to have occasional fails and a model that is known to be adequate, like in technical production/quality control processes. I doubt that it is a good strategy to use robust models in research, where your aim is to learn from data (also -and often even more- from "strange" and unpleasant data) rather than to control some process.
When one is dealing with time-to-event data (e.g., response times, saccade latencies, fixation durations, etc.) outliers can be identified by the shape of the RT distribution, as shown here: https://rdcu.be/b4n4g
This paper also explains why trials without an observed response time during the data collection period (or trials with an unusually long response time) should be treated as right-censored observations.
- Badges
- Science topic