Can someone suggest statistical methods to exclude outliers?

More Ivan Kadurin's questions See All

Does calculating the interaction index between a drug and a food matrix make sense?

Dear all, I'm conducting the revisions of some of my manuscript, with is the impact of the combination of Aspirin and a fruit juice in colorectal cancer cells. Two of the reviewers have asked to...

16 February 2021 3,141 3 View

Is it possible to full-couple the interFoam solver with the chtMultiRegionFoam solver?

We need to calculate multiphase stream of the one substance in the air atmosphere (this is why we using interFoam), but also we need to get heat transfer data between the substance, solid obstacle...

11 January 2021 872 2 View

Does anyone have experience with Land use/land cover data extraction from Second military survey maps?

I am trying to create polygon shapefile with land use from Second military survey using Arcmap. However, I really have hard time with reading the map, more correctly with boundaries between...

07 November 2020 6,018 3 View

How do I remove Bacillus sp. spores from Kraft Paper machine?

In the pulp and paper company that I work in, samples from paper, water and pulp from the paper machine gave information about existing spores of Presumptive Bacillus cereus. Our paper is used...

03 November 2020 1,726 10 View

What volume of a cell lysis buffer does one use for cells?

I have certain cells that are contaminant cells in a suspension of cells. I found a buffer that selectively removes them, but a lot of protocols don't specify what volume one should use for that...

03 November 2020 6,330 3 View

Best protocol to perform Western Blot from mice brain?

I am trying to look at the expression of c-fos nuclear protein in mice brain using WB technique. Can anyone suggest a protocol to extract the proteins? Better harsh homogenate before sonication?...

27 October 2020 5,272 2 View

Checking purity of sperm suspension?

I have sperm cells extracted from epididymis of a mouse. I would like to check and quantify their purity. My first idea is to use an intercalating dye like Hoechst and then see what fraction of...

25 October 2020 8,600 1 View

What do you think now of virtual teaching compared to face-to-face teaching of finished transmission of knowledge?

21 September 2020 4,178 10 View

How to store W/O droplets for over 10 days using paraffin oil with Span 80?

I am currently using a PDMS chip to produce water droplets with a diameter of 200 um in paraffin oil with 3wt%. After a couple of hours, the stationary droplets stored in the chip starts to form...

18 August 2020 2,459 3 View

How do you use factor analysis on categorical variables?

Hi, I'm trying to identify how similar different individuals are in their body traits. Normally, I would use a PCA or DFA, depending on my question. However, I have only categorical variables...

20 July 2020 4,171 1 View

Plant research: Watering uniformity/consistency. How important is it?

Trying to gather insight from an unresolved discussion with a colleague here, and either the available literature surrounding this topic and/or my keywords in search of are lacking. Context:...

21 February 2021 6,389 3 View

MVN, how to conciliate contradictory results from Multivariate normality and Multivariate outliers test?

Dear colleagues, I'm working with Path analyses in lavaan and MVN packages. There are some results that for me are confusing and features from the MVN that I do not know how to set. My dataset is...

18 February 2021 190 3 View

How to compare means of continuous data with zero-inflation and skew?

I have two groups (unpaired) of plant biomass measurements and want to know whether there is a test that can be used to determine statistical significance where assumptions are not violated. The...

18 February 2021 1,258 4 View

Why are the modelled and circuit system different in response for controlled buck converter?

[Images are attached at bottom] I am currently working on a buck converter controlled by two different control methods, Reinforcement learning applied to find PID values & a lead-lag...

16 February 2021 5,060 10 View

Mixed ANOVA with some normal and some non-normal data?

I am looking to run a 2x5 Mixed ANOVA to examine Sensory Processing Sensitivity in Highly Sensitive People (HSP) and Non_HSP. My Between-Subjects factor is HSP and Non-HSP. All participants tested...

11 February 2021 1,445 6 View

MAD or boxplot to detect outliers in non normally distributed data for latency and distance in animal cognition?

I am studying animal cognition and I have data about animals' latency to approach a cue and animals' distance from the cue in a test. Data are not normally distributed and when I do scatterplots...

04 February 2021 503 3 View

How do I report influence outliers in a pearsons correlation?

Good morning, I am running a correlation on phantom pain levels and self esteem scores in lower limb amputees. initially the data was reported as not normally distributed with the pain scale...

01 February 2021 600 6 View

Can you exclude participants from analysis if the time they took to finish their survey was too short?

Grezski et al., (2015) state that extremely short response times show a low data quality through a lack of attention of the participants, I used a cut-off point of 150 seconds based on an...

21 January 2021 7,468 5 View

What statistical analyses can I use for non-normal data overtly recorded with a 3 point scale and underlyingly recorded along a 0-100 scale?

I am struggling to understand what method of statistical analysis would work best for my data. I have a small sample of repeated measures data (15 participants in total) and I've asked them to...

09 January 2021 8,118 4 View

In case of larger sample size (more than 700 sample) can I do t-test though the data is not normally distributed?

My sample size is more than 750 but sk test is showing non-normal distribution. The variables have several outliers but I shouldn't remove those. In that case what should I do? t-test or other...

06 December 2020 1,285 19 View

Ma'Mon Abu Hammad

Certainly, there are several statistically tested methods for identifying and excluding outliers from a dataset. Here are some common approaches along with links to resources for further instructions and clarifications:

Z-Score or Standard Deviation Method: Calculate the z-score for each data point, which represents how far each point is from the mean in terms of standard deviations. Data points with z-scores beyond a certain threshold (e.g., 2 or 3 standard deviations) are considered outliers and can be excluded. Resources: Z-Score and Outliers

Modified Z-Score: This is a variation of the standard z-score method that uses the median and median absolute deviation (MAD) for calculations instead of the mean and standard deviation. It's less affected by extreme values and is suitable for non-normally distributed data. Resources: Modified Z-Score

Interquartile Range (IQR) Method: Calculate the IQR (the range between the first quartile and the third quartile) and identify data points outside a certain multiple of the IQR as outliers.Resources:Interquartile Range (IQR) Method

Tukey's Fences: Similar to the IQR method, Tukey's fences involve setting thresholds based on the IQR. Points beyond these thresholds are flagged as potential outliers. Resources: Tukey's Fences

Grubbs' Test: A formal statistical test that determines if a single data point is an outlier. It calculates a test statistic and compares it to critical values from the t-distribution. Resources: Grubbs' Test

Dixon's Test: A method for identifying a single outlier in a dataset. It uses ratios of differences between values to detect an outlier. Resources: Dixon's Test

Mahalanobis Distance: A multivariate method that calculates the distance between each data point and the mean of the dataset. Data points with high Mahalanobis distances are considered outliers.Resources:Mahalanobis Distance

Remember that the choice of method depends on the nature of your data, your specific goals, and any assumptions you're making about your data distribution. It's also important to consider the potential impact of excluding outliers on your analysis and results. Always document and justify your outlier removal process.

Koen Van de Moortel

Whether a measurement is an "outlier" depends totally on the model you use. A point may be far a way from the "best fitting" line through your data, but it may be close to an exponential function through your data.

The only reason to remove outliers I know, is if they are really wrong mesurements. A visual inspection of a scatterplot may suggest which points could be bad, although those can also be in the middle of your data cloud...

Jochen Wilhelm

Statistical methods can help you to identify points that are "far off from the rest". If these values are should or should not be excluded is not a statistical question but a subject-matter question. More often than not in research such "outliers" are desperately trying to tell you that your assumptions are bad. Excluding such values just make your wrong assumptions look better but your conclusions more wrong.

Generally, outliers should be rare. If they are, then they are usually not a problem in your analysis. If they are not rare, your whole experiment/assay is in doubt.

Carlos Jimenez-Gallardo

There is no statistical method that excludes outliers. It is a matter of Decision based on the background of the respective data. It is easy if you detect that it is a single case or a measurement error, but if it is real data, the decision is difficult.

advice, change the question, determine exclusion criteria.

For example, in Chile land of earthquakes, which is considered an atypical value. according to the measurements any over 8.0, and reaching the 9.5, however these are values that appear, but the 9.5 only appears 1 every 60 or 70 years maybe more, but those of grade 8 are more frequent in time, then what decision do you make?

so, once again, I agree with @Jochen