The real world data (e.g.environmental, geochemical studies), sometimes, do not follow a normal distribution. The data are found to be skewed and statistical analyses yield untrue results.
The log transformation, a popular method, is often used to transform skewed data to approximately “normal” and thus, to augment the reliability of the related statistical analyses. The log transformation tool is also found in many statistical software packages (e.g. SPSS).
What are the suggestions of statisticians, environmentalists and geochemists?
At least in psychometrics, where the reason for skewness is often related to the test being too hard or too easy, one can assume that the underlying cognitive ability is normal, within the population. So, if you think the lack of normality is due to the measurement method, and not the measure, transformation is justified on more philosophical grounds. However, when you use a statistical model, you are dealing with how this model sees the world and its assumptions for accurate performance, not the world of measurements themselves. Thus, throwing in a highly skewed variable in linear regression, is asking it to do something it cannot: to deliver accurate estimates in the presence of outliers, heterogeneity of variance and lack of multivariate normality. Logistic regression is actually less sensitive to distributional problems, but is sensitive to outliers. If you plot a variable against its log, or any other power-transformation, you will see that the relationship between them is non-linear. Thus, when you transform a variable to its log, you are very often linearizing the non-linear relationship between the outcome and your independent variable. Linear relations are more reproducible and understandible, even though several are not (perhaps most), than say, writing your own non-linear function. You can easily check this by plotting the untransformed and transformed versions against the outcome. The most common effect in a regression is that you get homogeneity of variance, multivariate normality and less outliers, mostly decreasing the risk of a type II error, but also type 1. The drawback is less interpretability of the effect size, because you really have a non-linear function, read out as standard beta. This can be partly resolved by simulation (clarify in Stata), or more simply, by graphing, or if your in luck, both the dependent and independent variables can be log-transformed, when beta is straight-forward on the log-scale. Still, some reduction in interpretability is arguably better then getting it completely wrong. If your dependent variable is non-normal, gamma-regression or counts (Poisson, negative binomial) is better, if they fit the distribution well. Don't log-transform counts.
In brief, in my view, the ideal is a model that is perfectly adept to the distributions that occur in the real world. This is unfortunatley not the case. Entering a highly non-normal variable into regression, is akin to pretending that a highly imperfect model is perfectly adept to the real world, which is naive. Using transformations is accepting lack of model perfection and dealing with the current data analytic reality, as it is, not as it could have been, but being aware of the limitations and using original scale, whenever its in line with model assumptions about the data.
Taking logs once will compress the top end of the data. Taking logs twice (RRSB/Weibull) compresses the data further. As would taking logs 5 times (RRRSB): doen't mean that this is a better or more realistic solution.... Always try to work with original model independent data - these are just mathematical models allowing further predicitions to be made. These may or may not be realistic and any selected model needs justification.
I found this article very helpful: http://www.jerrydallal.com/lhsp/logs.htm
All cases cannot be covered by a universal PDF. Thus some introductory inspection is suggested, for at least a qualitative characteristics of the investigated. For instance one can easily verify if the distribution is one- or multi-modal, a little bit more difficult is to state if the tail is or is not of heavy-type (with density decreasing with some (obviously negative) power of the absolute value of the observed random variable). Another way, suggested already by my preceder is to have a model of the distribution folloed for instance from ourunderstanding of the source/causes of the randomness. As a matter of fact, this is an art to choose the appropriate type of the distribution. Some experience can be reached by solving problems of the the form: given the probability distribution of the rv X, fid the probability distribution of the rv Y=g(X), where g is one of commonly used elementary functions. The above link illustrates the analysis for the case where X is normally distributed, basically with mean 0 and g=log. A more advanced problems are related to multidimensional X mapped by g into multidimensional Y . Examplary case is given as follows: if U and V are independent uniformly distributed over the interval I=[0,1], then the rv-s \sqrt{-log( 2*U) }*cos(\pi V) and \sqrt{ -log(2U) }*sin(\pi V) form a pair of iid normal standard rv-s. After establishing the type of the distribution, another problem is to choose the class of pd. An example: If we can accept that the rv is positive, with tail of power type, and the maximum is close to the origin - an appropriate class is build by Pareto-type distributions, be it with integer or fractional exponent, e.g. with 1 - F(x) = 1/( 1+ x^p), where p >0, for x>0. (here F stands for the cdf F(x) = Pr{ X
At least in psychometrics, where the reason for skewness is often related to the test being too hard or too easy, one can assume that the underlying cognitive ability is normal, within the population. So, if you think the lack of normality is due to the measurement method, and not the measure, transformation is justified on more philosophical grounds. However, when you use a statistical model, you are dealing with how this model sees the world and its assumptions for accurate performance, not the world of measurements themselves. Thus, throwing in a highly skewed variable in linear regression, is asking it to do something it cannot: to deliver accurate estimates in the presence of outliers, heterogeneity of variance and lack of multivariate normality. Logistic regression is actually less sensitive to distributional problems, but is sensitive to outliers. If you plot a variable against its log, or any other power-transformation, you will see that the relationship between them is non-linear. Thus, when you transform a variable to its log, you are very often linearizing the non-linear relationship between the outcome and your independent variable. Linear relations are more reproducible and understandible, even though several are not (perhaps most), than say, writing your own non-linear function. You can easily check this by plotting the untransformed and transformed versions against the outcome. The most common effect in a regression is that you get homogeneity of variance, multivariate normality and less outliers, mostly decreasing the risk of a type II error, but also type 1. The drawback is less interpretability of the effect size, because you really have a non-linear function, read out as standard beta. This can be partly resolved by simulation (clarify in Stata), or more simply, by graphing, or if your in luck, both the dependent and independent variables can be log-transformed, when beta is straight-forward on the log-scale. Still, some reduction in interpretability is arguably better then getting it completely wrong. If your dependent variable is non-normal, gamma-regression or counts (Poisson, negative binomial) is better, if they fit the distribution well. Don't log-transform counts.
In brief, in my view, the ideal is a model that is perfectly adept to the distributions that occur in the real world. This is unfortunatley not the case. Entering a highly non-normal variable into regression, is akin to pretending that a highly imperfect model is perfectly adept to the real world, which is naive. Using transformations is accepting lack of model perfection and dealing with the current data analytic reality, as it is, not as it could have been, but being aware of the limitations and using original scale, whenever its in line with model assumptions about the data.
Dr. Alan F Rawle,
Expert comment on the topic. The article provides an overview on "Logarithms". It is helpful to understand logarithms. Researchers will be benefited.
Thanks and regards
Dr Joachim Domsta and Mr. Lasse Giil,
Firstly, I thank you for addressing the question. You have profoundly tried to explain the matter. These deserve appreciations. I will go through all answers in detail. My best regards to you. Thanks
I often try log on biological data. I have never tried multiple log transformations. Most people ask that I provide raw values because log scale is too hard to understand. A multi-log scale might not go over well.
The log scale doesn't work well with negative numbers. You can fiddle with the result if you have to add a small constant to make sure that all values are greater than zero. If you add 1, or 0.1, or 0.0000001 you get a different answer in the statistical model. I try to select one value that is roughly the largest level that my experiment/equipment cannot detect. So if I use a balance accurate to 0.001 grams, then zero represents 0.0001 grams, or maybe 0.0004. The log transformation compresses both tails, the upper tail a bit more than the lower tail.
The other question is are you using the log transformation to solve a problem with the data, or because there is a theoretical model being built. If you have something like exponential decay, taking a log transformation is easily justified. If you are just solving a statistical problem, then you also need to ask if the need for the transformation is trying to tell you something fundamental about the system you are studying.
Dr. Timothy A Ebert,
A lucid description - useful information. Thanks and regards
Some environmental parameters follow an extreme value distribution. So another approach is to try fitting the data with different distribution types, see https://en.wikipedia.org/wiki/Generalized_extreme_value_distribution
It is justified to apply log transformation for the acceptable approximate transformation of non-Gaussian data into Gaussian. However, log transformation leads to the acceptable result in particular cases only. In addition, this transformation is not bijective. Therefore, in most cases for normalizing the non-Gaussian data it is better to use other normalizing transformations, for example, Su family of the Johnson translation system. For example, follow the links below.
Conference Paper Detecting bivariate outliers on the basis of normalizing tra...
Conference Paper Multivariate Outlier Detection Technique Based on Normalizin...
Of course, the logarithm is bijective if restricted in the domain to all positive numbers and in the image to all real numbers. Thus the problem is not in the non-bijectivity but in the range of values of the considered random variables! namely threfore - for instance - the log-normal is not applicable (withou modification) to observables suspected for having possibility to be negative!
Let me add to this that we do not have to express applicability of the log-normal to some not-well determined types. Instead we have to state clearly, that tis it perfect for such variables for which the logarithm is fitted by normal pd. And this type of STRICT determination of the range of applicability of families of distribution functions is always possible in terms of another families.
E.g. be it Weibull with cpd-sf given by 1-exp(-a*xp). It is a perfect family for random variables which are multiplicities powers of logarithm of random variables uniformly distributed over [0,1]:
Pr{ b [-log(U)]q >= x } = Pr{ [-log(U)]q >= x/b } = Pr{ [-log(U)] >= (x/b)1/q } = Pr{ U 0 and m2, v2>0, respectively, extended by the 5th unknown parameter p \in [0,1].
Regards, Joachim
Dr. Sergiy,
The attached papers regarding other normalizing transformations of non-Gaussian data, are educative.
My questions, "....log transformation leads to the acceptable result in particular cases only". Can you give some examples of suitable cases".
"......transformation is not bijective". What's the problem if there is no bijection?
Regards
Dr. Trimothy - Box-Cox transformation is suitable to reduce the skewness of a distribution? Regards
Yes. The Box-Cox transformation is a power transformation. If the best exponent is zero, a log transformation is used. An exponent of 1/2 is the squareroot transformation. I have not used this a great deal. I think all data values must be greater than zero.
Good: the method insures the best possible fit.
Bad: It may make results less comparable. I run an experiment with 20 replicates in each treatment. I run Box-Cox and get an exponent of 1.385. You run the same experiment in your lab, and in analysis the ideal exponent for your data is 0.975. Our qualitative answer should be the same, but can we compare quantitative answers? I think this will become more difficult. Of course the best answer is to share raw data, and all such problems disappear.
Such data as the ordinates of non-Gaussian stochastic processes (for example, extreme sea waves, speech signals, non-linear motion of ships) can not be normalized (satisfactorily be transformed into Gaussian data) by a log transformation. By the way, these data can not be normalized by the Box-Cox transformation too. We normalize such data using the Su family of the Johnson translation system.
Other examples relate to the estimation of confidence and prediction intervals for nonlinear regressions. So, for example, due to poor normalization of the data by log transformation, the width of the prediction interval of nonlinear regression is much larger than in the case of the Johnson translation system. Such example for the bivariate non-Gaussian data (actual effort in hours and size in adjusted function points from 133 maintenance and development software projects) you can find in the article (follow the link below).
As for the second question. If the transformation is not bijective, this does not lead to a one-to-one mapping of the set of non-Gaussian data to a set of Gaussian data, which generally leads to a distortion of the result (especially on the boundaries of the set), which is based on the application of such transformations.
Regards, Sergiy
Article CONSTRUCTING NON-LINEAR REGRESSION EQUATIONS ON THE BASIS OF...
Dr.Sergiy Prykhodko,
Specific comments on the questions. I appreciate your explanations. Thanks and regards
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis