Hi,
I have data that is based on sample of children and adolescents aged between 10 to 14. When I checked my IVs (scores on few different tests) and my depended variable (score on a personality test) for normality they all appear to be non normal according to Shapiro-Wilk test though skewness and kurtosis values were pretty ok for most of them (smaller than 1 and close to 0.5). I tried to normalize the data (which include relatively many zero values that I thought were behind that non normality) using log10, inverse and sqr root methods (after adding 1 to all values). Non of them led the data to normality. I didn't try reflection because I fail to see how that may help and also I didn't try box cox because it only applies for the DV.
I then tried to see the distribution when I divided the sample per age and per sex and the non normality gone. It seems that the strongest effect was of age. Meaning, my sample actually include three different populations- I didn't plan that, such significant differences were not predicted per age and sex.
The problem is that I really need to use regression and my sample size is not big enough to use only part of the data for that (I've nearly 300 participants but if you check them per age group and sex then we get six small groups of different sizes and considering I've 3 IVs, it's not a good idea). Is there anyway to normalize such data without dividing it to groups?
You state that you have three populations by age and possibly six by age and sex. You indicate that the groupings remove the non-normality. How distinct are the populations? If you plot the total data as a normal probability plot, do the populations form distinct groups? Natural groupings indicate that the groups should be analyzed separately.
Plot the data by groups as a probability plot and observe the slope. Are the groups sufficiently different? Does number in each group differ greatly? You should consider collecting more data by group.
Perform the regression with the assumption that the population is normal. What do you learn from the regression? Perform other types of regression as Jochen suggested. What do you learn?
What does it take to answer the question you asked before collecting the data?
Hi all, I agree with Jochen Wilhelm that a binary dependent variable and a logit model might be an appropriate approach to handling your data.
see http://www.itl.nist.gov/div898/handbook/pmc/section5/pmc52.htm and references
If your data are asymmetrical, you can try adjust skew-normal (library sn in R) or skew scale mixtures of normal distributions (library smsn or ssmn in R).
I guess I will take regression that doesn't require assumption of normality, Lasso or elastic net. What do you think?
Jochen,
regularized regression models are many times fit for both parametric and non parametric tests. They make a biased estimate of regression parameters and they don't have a definite rule for selecting the values of these parameters. Optimal value is found through cross-validation procedure, so to my best of understanding they don't make the assumption of normal distribution (though I don't fully understand the mathematical model behind them and they are new (and exciting) methods to me).
In my understanding also robust regression, which is not regularized technique is robust for violations of the assumption of the OLS regression models.
here is a new proposal
Article TO DETERMINE SKEWNESS, MEAN AND DEVIATION WITH A NEW APPROAC...
There are two ways to go about analyzing the non-normal data. Either use the non-parametric tests, which do not assume normality or transform the data using an appropriate function, forcing it to fit normal distribution. Several tests are robust to the assumption of normality such as t-test, ANOVA, Regression and DOE.
Transform a non-normal distribution into an approximately normal one,
Use the linear model,
Change the data,
Log transform
Too many extreme values in a data set will result in a skewed distribution. Normality of data can be achieved by cleaning the data. This involves determining measurement errors, data-entry errors and outliers, and removing them from the data for valid reasons.
- Badges
- Science topic
- Similar topics
- Social Science
- Sociology