Can an outlier test be run more than one time on the same data? For example: I run it once and it showed two outliers. I removed them and ran another outlier test. It showed another outlier. Can I remove it and keep running outlier tests until there are no more outliers or should I run it only once and delete the first run's outliers only?
I wonder what that fuss about "robust methods" is about.
If I have small samples, say a group size of n=5, and there is seemingly one value "outlying", then this represents already 20% of all the data I know. If I would consider this as being "strange" or suspect that something might by "bad" with this value - just because it's relatively far off from the other 4 values(!) - then I would declare that about 20% of my data are not trustworthy. That shouldn't be a basis for a confident interpretation of the results (however they might look).
Clearly, in small samples, one outlying value is influential. It adds a considerable amount of information, possibly very relevant information. That information is just not adequately considered by "robust methods".
Having large samples and 20% outliers would be like a hit in the face for anyone looking at it. No reviewer would be willing to accept a method or study (experiment, assay, survey) where one would dismiss 20% of the data just because it doesn't seem to "fit well" (to what, actually? to the expectations of the researcher ("data should noot vary much")?). But rare outlying values are not very influential in large samples - so there is no problem here to ignore them (to treat them like any other value). There is no need for special "robust methods".
Maybe my point of view is too narrow, I would be grateful if one could point out where the scientific* advantage of robust method is.
---
* that's different for process automatation, where ther is a possibility that something goes wrong and the mistake or problem is not recognized (because it's automated). Such instances can give extremely wrong values, distorting the analysis. Here one can employ an algorithm for "outlier identification and removal" or simply use robust methods. In science usually the experiments should be done carefully and the experimenter should see when things go wrong and not use values from failed measurements anyway!
If you want to identify outliers to select those as interesting candidates for further studies, then: no.
If you want to use the outlier test to remove data from another analysis, then you should reconsider doing outlier tests at all. Concluding that values that had some "non-significant" p-value in an outlier test are not outliers is flawed logic ("thou should never interpret non-significant results!"). Outliers should be identified through *external* reasons, not "internally" by looking at their p-values in outlier-tests. If you have more than about 1% outliers, I would be sceptical if either the experiment is ok or the assumptions I have about the distribution of the data are adequate (or both).
If you do a quality control screening in a process, you may optimize the expected win or loss. If larger deviations in the output are costly, it may be advantagous doing several rounds of removing outliers.
My guidance on excluding outliers would be to be careful using a statistical test (and I've published on means of outlier detection). Outliers, as I understand the term, refer to observations within an experiment that are not commensurate with the other observations (or technically, the result of the same data generation process) and should thus be excluded.
Suppose you want to evaluate the effects of a medication adherence outreach and there are some hemophilia patients in the analysis. If your hypothesis is, what is the effect of this outreach on the average customer, because these patients have orders of magnitude higher cost than other patients they can be regarded as outside the purview of the experiment and hence outliers. But if you had one oncology patient with unusually high cost you might need to retain that observation in the analysis because they contain real information about the effects of the outreach.
In other words, outlier tests are useful in identifying potential outliers but you need to have a clear understanding of the purpose of the experiment to know if outliers truly belong with the other observations or need to be excluded to best answer the original hypothesis.
Notice that because of masking, sequential application of the test for a single outlier for detection of multiple outliers can lead to detection errors and is not recommended. See article 'Algorithm 974: The OutlierLib - A MATLAB library for outliers' detection' , ACM Transactions on Mathematical Software, 43(4):38, March 2017 (https://www.researchgate.net/publication/315519154_Algorithm_974_The_OutlierLib_-_A_MATLAB_library_for_outliers%27_detection)
Article Algorithm 974: The OutlierLib - A MATLAB library for outlier...
the outliers presence can to be, for information lack, traduced in lack of variables. Analyze the nature of outlier
When dealing with outliers, its difficult to recognize them by the size of the residual o error since they can be small for outliers. One way of dealing with outliers is not removing them but using a robust method that deals with the presence of outliers. This way you can learn about what observations are considered outliers and by not removing them you publication becomes stronger.
I wonder what that fuss about "robust methods" is about.
If I have small samples, say a group size of n=5, and there is seemingly one value "outlying", then this represents already 20% of all the data I know. If I would consider this as being "strange" or suspect that something might by "bad" with this value - just because it's relatively far off from the other 4 values(!) - then I would declare that about 20% of my data are not trustworthy. That shouldn't be a basis for a confident interpretation of the results (however they might look).
Clearly, in small samples, one outlying value is influential. It adds a considerable amount of information, possibly very relevant information. That information is just not adequately considered by "robust methods".
Having large samples and 20% outliers would be like a hit in the face for anyone looking at it. No reviewer would be willing to accept a method or study (experiment, assay, survey) where one would dismiss 20% of the data just because it doesn't seem to "fit well" (to what, actually? to the expectations of the researcher ("data should noot vary much")?). But rare outlying values are not very influential in large samples - so there is no problem here to ignore them (to treat them like any other value). There is no need for special "robust methods".
Maybe my point of view is too narrow, I would be grateful if one could point out where the scientific* advantage of robust method is.
---
* that's different for process automatation, where ther is a possibility that something goes wrong and the mistake or problem is not recognized (because it's automated). Such instances can give extremely wrong values, distorting the analysis. Here one can employ an algorithm for "outlier identification and removal" or simply use robust methods. In science usually the experiments should be done carefully and the experimenter should see when things go wrong and not use values from failed measurements anyway!
I would not totally agree with Jochen, since the "standard" methods, like t-tests asses means, and the finite sample breakdown point of the mean is 1/n, which is very small as compared to the median or trimmed estimates. Therefore, even in larger (what is large) samples, single values can be very influential and should handled cautiously (as already mentioned by others). From this point of view, robust methods are needed.
But I agree with Jochen in so far, that this would not so much of an issue if the distribution of the data would be explicitly taken into account to chose an appropriate statistical model instead of "always" using regression models for example, which imply a priori a specific distribution of the data/residuals. In so far, robust methods are not needed.
;-)
An outlier test may be performed by the applicability domain of the developed model (i.e., to identify the test set/query compounds those are outside the applicability domain and also detect outliers from training set compounds).
Applicability domain (AD) is the “physicochemical, structural or biological space, knowledge or information on which the training set of the model has been developed”.
The Euclidean based applicability domain is very useful to detect the outliers: The resulting model can be reliably applicable for only those compounds which are inside this domain. It is based on mean distance scores calculated by the Euclidean distance norms.
Euclidean distance-based Applicability domain is provided in http://dtclab.webs.com/software-tools
Moreover, You may also go through these articles:
1. Roy, Kunal, Supratik Kar, and Pravin Ambure. On a simple approach for determining applicability domain of QSAR models. Chemometr Intell Lab Syst. 145 (2015): 22-29.
2. Golmohammadi, H.; Dashtbozorgi, Z.; Acree, W. E., Quantitative structure-activity relationship prediction of blood-to-brain partitioning behavior using support vector machine. Eur. J. Pharm. Sci. 2012, 47, (2), 421-429.
3. Amin, S. A., Adhikari, N., Jha, T., & Gayen, S. (2016a). First molecular modeling report on novel arylpyrimidine kynurenine monooxygenase inhibitors through multi-QSAR analysis against Huntington’s disease: A proposal to chemists!. Bioorganic and Medicinal Chemistry Letters, 26, 5712–5718. doi:10.1016/j.bmcl.2016.10.058
4. Amin, S. A., Adhikari, N., Gayen, S., & Jha, T. (2017). First report on the structural exploration and prediction of new BPTES analogs as glutaminase inhibitors. Journal of Molecular Structure, 1143, 49–64. doi:10.1016/j.molstruc.2017.04.020
5. Amin, S. A., Adhikari, N., Gayen, S., & Jha, T. (2017c). An integrated ligand-based modelling approach to explore the structure-property relationships of influenza endonuclease inhibitors. Structural Chemistry, 28, 1663-1678. doi:10.1007/s11224-017-0933-z.
Don't forget that all outlier test rely on the normality assumption, or a similar one. If you come up to remove several outliers, and that removing outliers makes other outliers to appear, then either your data are of a so bad quality that you should just not use them at all, or your normality assumption simply does not hold and you must search for other distribution models or use suited statistical methods to handle non-normality.
So the quick answer: your procedure is deeply, definitly, incorrect.
- Badges
- Science topic