Hi Elizabeth,

You should proceed with caution when considering to remove observations from the data. In many cases, there is a valid reason for these observations to be outliers and that is what the researcher should be studying. Why was this an outlier?

Another issue with outliers is where to draw the line. It may not be clear where the outlier behavior starts. There are some people who arbitrarily eliminate a percentage at the tails (e.g. 5%), which makes no sense whatsoever.

Finally, you should not take out the outliers and then transform the data. The data may appear non-normally distributed because of those data points. So eliminating them may in fact cause the data to appear normally distributed. So by transforming the data, you didn't improve the fit.

Again, you should think carefully about what is an outlier, why is it an outlier, and what should you do with the data if you choose to keep the outliers in the data.

I hope this helps

Ariel

If your goal is to transform the data so that the resulting distribution of the transformed data is approximately normal, you may use the following approach without removing any observations: First compute the empirical cdf (e.g., in R use ecdf) and smooth it by using say linear or quadratic splines and call the smooth estimate of the CDF, G(X). If X has a pdf, then U=G(X) will be approximately uniform distribution and so Y=Q(U) will have an approximately normal distribution, where Q denotes the quantile function of a N(0, 1) variable. Here is a sample R code (just copy and paste the following lines in the R window):

#####Generic Transformation to normality#######################

x=c(rnorm(25,-1.5,0.5),rnorm(75,1,0.5)) #data from mixture of normals

par(mfrow=c(2,2))

hist(x,prob=T) #you will see a bimodal shape

Fn=ecdf(x); fit=smooth.spline(x,Fn(x))

plot(fit,type="l")

u=predict(fit)$y

hist(u,prob=T) #you will see uniform shape

z=qnorm(u)

hist(z,prob=T) #you will see a normal shape

lines(sort(z),dnorm(sort(z)))

ks.test(z,"pnorm") #formally tests if transformed data is normal

######Copyright(2015): Sujit K. Ghosh################

A sample output from above code is attached

I suggest that the data be transformed to normal quantiles first. Then use regular parametric methods on the normal scores.

Thank you for your answers!

Maybe an add-on to Ariels excellent answer:

Note that "outliers" are rare. If they are not rare, they are either no outliers or your method/data collection is seriousely flawed (and son on might not really trust in any analysis at all).

As Ariel stated, there should be some "external reason" to judge if a value is an outlier. It may be known that something went wrong with the measurement or the sample, or the value may be "unphysiological" (like a blood systolic pressure of 417 mmHg - in a living person...) or clearly impossible (like a negative blood pressure, or a relative humidity of 120%, thinks like that). Fur sure such value are "real outliers" and MUST be excluded (it would be stupid not to exclude them).

It usually makes no sense to judge outliers as such based only on the available data. Outliers are rare, and the will be "visible" from data only if there is a considerable amount of data. In small data sets, "outlying values" may be just the only values "turning things right", giving you the most important information, e.g. that you would underestimate the variance if you used only the other data, or that the distributional model you assume does not fit the data. But if there is much data available, so that outliers could be recognized as such, their impact on the analysis will usually be negligible. So why care?

To my experience, "outliers" are frequently the most important points and they should NOT be removed (unless there are external reasons!). They should rather provoke to re-think your model. If you think your model is appropriate, then such outliers will save you from underestimating the variance (in smaller samples) or they will be negligible (in larger samples).

Exactly. Therefore, using normal quantiles in place of the raw data keeps any extreme values in place, but their roles are smaller than with the raw data set. An outlier can be a data value that is a measurement error, or maybe a value that has been recorded incorrectly, or maybe a value that happens to be far away from the center of the data from a population with high variability, or it is a value that appears "far out" but it has not much "leverage" or influence on the analysis. Such cases often appear lumped together under the name "outlier". If you are not able to figure out what these extreme values really are, then using a robust approach may be a good way to go.

I have several articles published on using "extreme values", and I explained in them that using statistics based on extreme values amplifies their roles in the data analysis, as compared with a robust analysis in which their roles are being downplayed. It all depends on what you are targeting with your data analysis.

As Jochen said, "there is information in the tails", as George Box has said.

Raid,

thank you for this: "As Jochen said, "there is information in the tails", as George Box has said." - I did not know this citation, but I really like it - it hits the mark.

:)

I think that the original idea is from Francis Bacon, who said that we learn once when we establish the general rule and a second time when we examine the exceptions.

But the whole question struck me as strange. Transforming data risks losing vital information about effect size. The only reasons I can think of for transforming data are ones that are based on science, rather than statistics – for example, we routinely transform bacterial counts in drinking water to log10 units. This makes sense in terms of the process by which bacteria multiply.

And my motto used to be I'm not an outlier; I just haven't found my distribution. In other words, what can appear as outliers may be observations driven by a separate process. I was working with some health costs data, and a clutch of strange low values turned out to be people who received a subsidy. Once understood, they are not outliers.

So think about the shape of your model, not your data. And think of the processes that generated your data. Understand those and you probably don't have to delete anything.

I agree 120% with Ronán (at least, take it as lower bound estimate).

I just want to stress that for biological/physiological data, a log-transformation makes pretty much sense (as Ronán explained for bacteria counts) in the majority of the cases, because of the underlying "multiplicative" processes that generate the data.

It depends... Look at the data points and decide whether they are outliers or outlying high leverage data points. In the instance of the former, removing them will have little effect on the overall results and you could transform the data afterwards--you will get a better fit (reduced bias). However, in the instance of the latter, you may drastically change the outcomes of the analysis--you land up removing really important data and then change the distribution of the data through transformation—you start taking a long walk down a twisting and turning garden path.

So, if you remove simple outliers that don't shape your results, you could justify computing a transform of the data afterwards if it improves the fit (reduces bias). Just be sure to understand the nature of the data and then proceed from there: test different scenarios, check mean squared errors (if you are not changing the complexity of the model), check the change in beta coefficients and/or F-ratios.

Great to read all your insights.  I am also in agreement that outlier removal has to be done after due deliberation.

At the same time, I am also interested to know whether the order of "outlier" removal and "data transformation" has any impact on the outcome.  I believe that was the question from which the whole thread started.

Might be helpful - https://www.researchgate.net/post/When_is_it_justifiable_to_exclude_outlier_data_points_from_statistical_analyses

Ronan, this is exactly my issue - which distribution is it? I have a series of payroll deposits, and I'm trying to determine the frequency (weekly, monthly, etc.) and a representative amount. However, the data includes (mostly) payroll amounts, but also expense checks (smaller), and bonuses (larger), three different populations. By eye it's easy to tell them apart, but tricky using statistics drawn from the series.

George Box (I think) said "there is information in the tails."  I agree with what Ariel has said. Treat "outliers" very carefully. 

perfect question. my answer is; NEVER remove outliers. try our new paper

Article TO DETERMINE SKEWNESS, MEAN AND DEVIATION WITH A NEW APPROAC...

Actually I have the same problem,

However in my opinion, I usually see what happens when I remove it. If it has a big impact on the results, then I will mention that, if it doesn't have a big impact, I also mention it. Either way, I prefer to keep the outlier in my data because removal is in my opinion a bias.

Moreover, first you need to try to figure out why the outlier is present!

It is important to figure out if a certain data value is a "true outlier" or a "influence or leverage point". Is it due to some measurement error or is it maybe due to excessive variability in the data? 

If the data are non-Gaussian and the outlier detection is carried out on the basis of the normalizing transformations, then the outliers be removed after data transformation. For example, follow the links below.

Conference Paper Detecting bivariate outliers on the basis of normalizing tra...

Conference Paper Multivariate Outlier Detection Technique Based on Normalizin...

There are three types for what the community refers to as outliers and they are used interchangeably:

Outliers: Typical data points that far away from the mean or median. e.g., a heart rate of 140 beats/min for a subject during resting condition. The subject might got a medical condition during the experiment.

Anomaly: A data point that is not legit due to error in data collection or device issue. e.g. a hear rate of subject of 10 beats/min.

Outliers-anomaly: a data point that is far away from the mean or median, but you cannot decide the source of it.

It is Okay to remove the anomaly data before the transformation. But for other cases, you have to have a reason for removing the outliers before the transformation. Unless you can justify it, you cannot remove it because it is far away from the group. Maybe it is the side effect of the experiment that should be taken into consideration.

It is not necessary to detect any outliers as long as you assume that the sample is obtained from a continuous distribution and use the data to estimate the "probability integral transform." Notice that if X has continuous cdf F, then Z=Q(F(X)) has a standard normal distribution where Q(u) denotes the quantile function of the standard normal distribution.

So, if the true underlying cdf is estimated by say empirical cdf or its smoothed version, then the above transformation can be estimated uniformly almost surely (asymptotically speaking). An illustration of the method appeared in one my my earlier posts:

Thus, if your sole goal is to transform a random sample of observations to normally distributed sample, see the approach discussed is another post:

https://www.researchgate.net/post/A_generic_transformation_to_normality

If your goal is the identification of outliers or making inference in the presence of potential outliers, you may find the following article useful:

Article Robust and Accurate Inference via a Mixture of Gaussian and ...

I have the same question. The most recommended answer doesn't really answer the question.