Thanks Sujit. I've been looking for generic transform to Normality methods for some time. This looks very good

I agree Sujit.  I just had the same discussion with someone at JSM.  I'm not sure, but I suspect an eCDF method will have smaller bias but perhaps larger variability, especially for small n.  For "larger" n, I would be fairly confident that this is the way to go. 

Sujit, why do you want to transform the data to normality?  I ask, because very often, people want to transform to normality when there is no need to do so.  E.g., the link below has several old usenet newsgroup posts on normality and linear regression.  HTH.

http://www.angelfire.com/wv/bwhomedir/notes/normality_and_regression.txt

Thank you for the information.

This method always work, it is «non-parametric» but is connected to a single sample. Sometimes, one needs parametric models across several samples, and doing predictions across possible future observed samples. See for example my work Mønness, E. 2011. The power-normal distribution: application to forest stands. Can. J. For. Res. 41(4): 707–714.

Also, the sample may include regression effects, and it is the residuals that should be normal: See E. Mønness (2012): Data Transformations with a Full 2^6 Experimental Design—A Metal-Cutting Case Study,Quality Engineering, 24:1, 37-48

Modelling should not be just generic. It should also be parsimonious. The normal distribution is parsimonious as it only involves two parameters. By adding the parameters of a spline to the model makes the modelling inflationary, ineffective and with low predictability. So, sorry, I would not use the proposed methods in modelling. I would try a transformation with one (or two at most) parameters, additional to those of the normal distribution.

Some papers related to the value of parsimony follow in the links below.

http://www.itia.ntua.gr/en/docinfo/1516/

http://www.itia.ntua.gr/en/docinfo/1343/

http://www.itia.ntua.gr/en/docinfo/1432/

Article Entropy: From Thermodynamics to Hydrology

Article Just two moments! A cautionary note against use of high-orde...

Modelling should not be just generic. It should also be parsimonious. The normal distribution is parsimonious as it only involves two parameters. By adding the parameters of a spline to the model makes the modelling inflationary, ineffective and with low predictability. So, sorry, I would not use the proposed methods in modelling. I would try a transformation with one (or two at most) parameters, additional to those of the normal distribution.

Some papers related to the value of parsimony follow in the links below.

http://www.itia.ntua.gr/en/docinfo/1516/

http://www.itia.ntua.gr/en/docinfo/1343/

http://www.itia.ntua.gr/en/docinfo/1432/

Article Entropy: From Thermodynamics to Hydrology

Article Just two moments! A cautionary note against use of high-orde...

Article Generic and parsimonious stochastic modelling for hydrology and beyond

Bruce Weaver is correct. It may sometimes be desirable to have errors or estimated residuals that are normally (or t-distributed), but in general it is not necessarily useful at all.  The central limit theorem, for example, is not about transforming the original data. And for regression, I found great usefulness in model-based estimation/prediction/and imputation where estimating the variance of prediction errors was important, and there it might be somewhat desirable to have normal-like random factors of the estimated residuals in WLS regression, but not so much, and the data for the establishment surveys with which I worked were highly skewed. That is the nature of the data, and there is nothing wrong with that.

But as Bruce implied, it depends upon your application and goals. 

Sujit,

In experimental design and we we want to build an efficient model for prediction, it is essential to obtain normal response distribution.

In regression modeling, it is necessary some times to obtain normal distribution for the response variable for better fit.

Box-Cox transformation may be a little easier to obtain normal distribution, as shown:

require(MASS)

library(MASS)

boxcox(lm(response ~ ., data =data name), lambda=seq(0,1,by=.1))

# then empirically and automatically estimate lambda

response.bc

Watheq,

"In experimental design and we we want to build an efficient model for prediction, it is essential to obtain normal response distribution." - That's just not correct, in several ways. The distribution of the response does not matter - the distribution of the residuals is the key, and this is tightly linked to the formulation of the model. And the predictions of models assuming normal residuals are BLUE, even when the actual residual distribution is non-normal. The important part here is the estimated uncertainty of the predictions, what depends on how the stochastic part of the model is defined. Note that this does not neccesarily need to be based on the normal distribution!

"In regression modeling, it is necessary some times to obtain normal distribution for the response variable for better fit." - Again it is not the distribution of the response variable that really matters, but the distribution of the residuals. Apart from this I do not see how this is related to the quality of a fit. How do you measure this quality? All "quality" measures I know depend on the scale of the response, so transforming the response also transforms the meaning of these quality measures, and comparing them is like comparing apples and peaches. At best, It is just meaningless.

A good model gives estimates of interpretable effect sizes. Most transformations will render the estimates quite uninterpretable, what I consider a major disadvantage (NB: the log-transformation is somewhat special because it just makes a model estimating multiplicative effects instead of additive effects).

Dr. Jochen,

What I know from my knowledge from regression analysis, if we fit our model and don't have all the predictors reject null hypothesis. Then, we should try to obtain normal response distribution and re-fit the model.

I have done that a a lot in Petrophysical modeling and tested the difference between non-normal and normal response distribution.

Yes, I agree the residual normality is essential, but as I said sometimes, we need to obtain normal response distribution prior to fit the linear regression.

The same thing for Experimental Design workflows.

Second, if I don't get normal response transformation, I rarely find normal residual distribution.

Best,

 Sujit:

Greetings.

In fact, this approach is kind of known and one of my PhD student is relying extensively on this approach for modelling all sorts of data although he tends on concentrate more on financial statistics problems. He is graduating this year or the next. We tend to rely on copula to do so.

In fact, you will find this approach used, although not advertised exclusively as such, in a few articles. Once such article that comes in mind is by Regina Liu on control charts (In Handbook of Statistics, Vol 22 Statistics in Industry Ed R. Khattree and C. R. Rao), where she mentions this in the passing.

Folks: Thanks for all the feedback and I appreciate all of your efforts to respond to my original query. In particular, Bruce (Weaver) asked a very relevant question, as why would we look for such a transformation.

I see that the discussion about this issue went towards transforming the response variable within a regression model of the form: y = m(x) + e where m(.) denotes the regression function of the vector x and e denotes error variable with mean zero. In such a scenario, we can perhaps use one of multivariate nonparametric regression tools (kernel, splines, etc.) to directly estimate the regression function without making any transformation to y (e.g., check out many tools available in the R package 'np'). Of course, as pointed out by various folks (within this post) that cleverly selected parametric transformation or models may perform as well, but of course we all know the famous quote by George Box: "...all models are wrong, but some are useful."

My goal for such a transformation was partly motivated for transforming the marginal distributions within a copula type models (as pointed out by Prof. Khattree). So, even in the context of regression models we could perhaps estimated the joint density of (y, x) by first transforming the marginals to normals (using the method I described in my original post) and then use copula to estimate the joint and then simply calculate E[y|x] from the estimated (smooth) joint density of (y, x).

I have used a similar method while developing a joint model for imputation in high dimensional missing data scenario (in which Josh was also involved, who has already responded within this post). In case, anyone is interested in our JASA-2013 paper, it is available in the attached link:

http://www.tandfonline.com/doi/abs/10.1080/01621459.2012.734158

Article Imputation in High Dimensional Economic Data as Applied to t...

The method described by the author is commonly applied in geostatistics and is known as the normal scores transform.  One  reason for the approach is that high speed simulation methods based on stationary random functions and the fast Fourier Transform are available for normally distributed data, which are subsequently transformed to the original distribution using the lookup table based on the transformation method described in this post.  One issue not discussed in this thread that is critical is the choice of a tail model to extrapolate beyond the highest and lowest percentiles.  This can be critical as the tail is heavily influential on the mean of the distribution.

See texts by Goovaerts  or Journel in Geostatistics.

Thank you John. This seems to make sense :)

Surely, to create a lookup table one needs to have a good estimate of the distribution of the original data, and that requires to have quite a bit of data. Before my inner eyes I see physicians with n=7 using the normal scores transform to be able to apply a t-test...

No no no...N>>>>100 and even then choice of the tail model can completely swing estimates of the mean...this is a sensitive game.  In Mining the mean grade (equivalently stock price)  matters, so these guys pay a lot of attention to this issue.

I understood that (I think). The tails remain a problem, the tail model should be resonable, and wrong assumptions about the tails can severly impact the conclusions. Again looking from the angle of a physician (as good as I think I can...) the data is often bounded (for instance there is a more or less physiological range of a blood pressure in humans, or a concentration of an analyte can only be between 1 molecule and the saturation point). So it should be possible - at least in principle - to get some sensible tail model. But on the other hand do physicians rarely use methods like FFT, so all my thoughts are likely a bit "misplaced" anyway...

Transformations made to do anything in the furtherance of hypothesis testing (say, to "remove" the naturally occurring heteroscedasticity) is worrisome to me (see attached link). 

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

The main reason I can see to avoid the normality transformation here is a commonly found one, described, at least partially, earlier by Jochen: 

"A good model gives estimates of interpretable effect sizes. Most transformations will render the estimates quite uninterpretable, what I consider a major disadvantage...." 

If a transformation can be avoided, I would. Otherwise, they may "muddy the water."

One resource, a book I read 'cover-to-cover,' a number of years ago, Transformations and Weighting in Regression(1988), by Carroll and Rupert, was quite interesting and well done, but I think transformations, in practice, are often overdone. That is not to say that the procedure here does not have merit, as others in this thread indicated, but I would hope it not to be used unnecessarily. 

Article Practical Interpretation of Hypothesis Tests - letter to the...

Looking back to the origin of this thread we see "Here is generic method to transform a random set of values to normality," and then later Sujit noted that "My goal for such a transformation was partly motivated for transforming the marginal distributions within a copula type models (as pointed out by Prof. Khattree)."       

So though regression residuals are more interesting to me, that is quite a different matter, and this was really originally exclusively about data sets or populations.   Correct?   I have not had reason to want to make such a transformation to normality there either, but different areas of statistics have different needs, so out of curiosity, I did a quick internet search to ever-so-slightly edify myself regarding the meaning of and uses for Gaussian copulas. 

I expect there are many good uses, but as in any other statistical method, there is potential for problematic applications.  Here is a link to a paper I just found,  on an interesting problem:

"GAUSSIAN COPULA, What happens when models fail?" by Forslund and Johansson, 2012

http://www.math.chalmers.se/~rootzen/finrisk/gr15_forslund_johansson_gaussian_cop.pdf

It is also attached in case this link fails. 

In the introduction, the authors note that "Using Gaussian copula based methods banks [were] able to bundle together high risk credits into CDOs that appeared to carry low risk," which apparently contributed to the 2008 economic collapse.   ???

Can anyone tell me/us if blaming a Gaussian copula in this case makes sense, why, and/or what misuse may have been made of this technique?  A simple statement of the basic idea would be appreciated. Thanks.  

Yes, Gaussian copula is known to be notorious as it can't adequately model the tail dependence when the underlying distribution is fat tailed. But there are many other alternative class of copula that can adequately address the tail problem in practice. The choice of a particular class of copula is perhaps still a matter or art and so in practice lot of care must be taken and results be (cross) validated.

James:

In finance literature, this "fiasco" is well known but I must say that (as is not uncommon in corporate life) the blame is misplaced. David X. Li's work was a theoretical attempt to model default probabilities, by borrowing the ideas from reliability theory and he drew parallel of  "default" to what is "component failure" in industrial setup. The fact which should have been obvious to practitioners and they apparently neglected, was that there are underlying assumptions which needed to be verified.  Li had used no data and had not given any illustrations. He simply used bivariate Gaussian copula to provide a theoretical possibility. Financial industry did not care to check the applicability and it soon became a popular (rather, more or less standard) approach for calculating the joint default probabilities even through it was quite well known that the associated probability distributions tend to be fatter tailed than normal, thereby severe underestimation of the default probabilities using Gaussian copula.

Clearly, we cannot blame a technique for failures if the person who applied it does not even understand it. It only goes to illustrate how important are the roles of trained statisticians and how things can backfire if  an untrained person blindly applies a techniques without understanding it. What is more ironic is the misplaced blame by those who were actually responsible for it. The situation is akin to a malpractice by an unlicensed physician, who then places the blame for death on the malfunction of the machine, even when he did not know how to operate it. 

Thank you Sujit and Ravi for the informative explanations.  

I'm old and forgetful, but this is sounding similar to work I did many years ago, where I did not transform distributions to normality, but instead used distributional forms closer to what was observed, or distribution-free statistics when necessary.  I think, as Jochen noted, that interpretability is extremely important.  And your point, Ravi, about people using statistics they do not understand is something that I think makes the miracle of modern software a little scary.  (Programming using FORTRAN 77 may have been crude, but at least you might have a better idea as to what you were doing - maybe.)  John's interesting comments on geostatistics touched on computing power in a different way.  At any rate, the "interface" (as one series of symposia is named) between computer science and statistics is important here, as the thread Sujit started here demonstrates.  I was a lousy programmer - not good with details - and eventually had other people doing it, but if you don't at least understand the software limitations, similarly to not understanding the foundations of the statistics used, things could go 'horribly wrong.' 

Thank you all for the interesting thread and information. 

I am interested in your question because I usually tell my students: "Transformation of a random variable to attain normality is not a good idea. The nature of the data, how data is scaled, how the deviations from data point to data point are measured, or which is the sample space and its structure, are points that tell you which transformations are meaningful. It does not matter if the resulting variables fit or not the assumptions of a standard model: you can not forget the assumptions on the sample space to screw your data to e.g. make the residuals normal in a regression". Reading the answers given to the question I fully agree with comments by Jochem Wilhelm and others about the need of transform variables into normal ones. I would like to refer to the sample space approach. I take a simple regression model for reasoning.

In a simple regression model Y=a+bX+e, residuals are computed as a subtraction e=Y-(a+bX) and, therefore, subtraction (-) and its opposite (addition, +) should have a clear meaning in the sample space of Y. Moreover, ||e||^2 is assumed meaningful, as it is minimized for an optimal least squares fitting. This means that Y and e are considered as real random variables. If some transformation T of Y is taken, for instance, in order to make residuals to be approximately normal, the residuals will take the form e'=T(Y)-(a'+b'X), or even more involved if X is also transformed. Again the new model implies that T(Y) and e' are real random variables. However, for most transformations T, both sets of assumptions are not compatible. For instance, the transformation changes the sum in the sample space of Y: initially the sum of two random variables, say Y_1, Y_2 was Z = Y_1+Y_2. After the transformation the associated addition will be W = T^{-1}[T(Y_1)+T(Y_2)]; note the implications on the way of computing mean values, distances and norms. The addition associated with T may be meaningful to the problem analysed or not, and this should be the criterion to select a transformation T, not whether the resulting distribution of the new residuals is normal or not.

Very often the variable Y is a positive variable and the zero and negative values are not possible results. The question is wether the ordinary sum and difference are relevant operations in the sample space of Y. In general, the predictor a+bX can take null or negative values, what is nonsensical for a consistent model. A way out is to take a transformation T that transforms the sample space of Y into the complete real line. For instance, T=log can do the work, provided that the scale of log(Y) is absolute and differences between log's are meaningful. This is not related with the fact that residuals are normally distributed or not.

My conclusion is that pre-existing models may not match a given problem. Transforming variables should be taken according to the scale and operations assumed relevant for the problem and not to fulfill conditions of the models at hand. This is a very strict methodological point of view; however, who has not used a square root of response in regression or a Gaussian anamorphosis in kriging (normal scoring)? At least, I did.

 Juan José Egozcue is right. In the field of compositional data analysis the recognition of the structure of the sample space as an Euclidean space (what we call the Aitchison geometry of the simplex) meant a big step forward, both for theoretical developments and for applications. Understanding that the sample space of the random variables and the assumptions of a model have to be the same, and I mean not only the set of possible values, but also the algebraic-geometric structure, was the key; and I think it is fundamental to any approach. 

In industrial designed experiments, the number of possible influencing X variables (often in 0/1 levels) is high, and the number of observations are low, often close to the number of observations. The residual are often small, since the Y is the result of a very specific production setup, but also since the regression is close to just a linear transformation of the observed data (due to number of variables close to number of observations.

Let y=f(Y) be the transformed variable, and f-1 is the inverse transformation . Let y^=X(X’X)-1X’y. With small residuals, y^ ≈y. Thus f-1(y^)≈f-1(f(y))=Y. So, on the original scale, transformations do not change the expected values much.

Transformation is then a tool to pick the right columns of X, get a simple model with few interactions.

The BoxCox should not have to high lambda values for this to be approximately true. High absolute values makes the transformation close to vertical or horizontal straight lines removing variation almost completely, and also the truncation issue of BoxCox pop up as a problem.

Thank you all for the fruitful discussion. I am coming from the field of civil engineering and my research is focusing on time series analysis of rainfall and also I use geo-statistics methods like Kriging. As far as these methods are concerned I always need to transform the data to Normal. Because these methods assume that the data is normal. So, if the data is not normal, I have to use one of the transformation techniques to be able to apply such method.  I was always wondering if I could release the assumption of normality and work on the data without transformation. But in these methods I can not do that.

Below is a link to one of my papers based on these transformations.

https://www.researchgate.net/publication/258225101_Geostatistical_analysis_using_GIS_for_mapping_groundwater_quality_case_study_in_the_recharge_area_of_Wadi_Usfan_western_Saudi_Arabia

Article Geostatistical analysis using GIS for mapping groundwater qu...