I analyzed the skewness and kurtosis of one of my dependent variables in my my data against the independent variable of 'gender' to get the z-values.
For males, the skewness z-value is +0.79 which is a little skewed, and the kurtosis z-value is +4.90 which is largely kurtotic!
For females, the skewness z-value is +3.19 which is largely skewed, and the kurtosis z-value is +1.16 which is little kurtotic.
Please what does this mean? How do I make a conclusion?
"Data" can never be normal; the normality assumption does *not* refer to the observed data. Rather, the assumption is that the *process* that produces the data is a normally distributed process. And that process, likewise, can never be precisely normal, because of asymmetries, discreteness, and boundedness of the observable data. So the question is never "normal or not". It is always "how different from normal is the distribution that produces the data?" When that distribution is close to normal, the methods should be fine. And, depending on the method and sample size, the methods may be ok even when the distribution is far from normal because of the Central Limit Theorem. There is no "one size fits all" answer, rather the answer is always, "it depends." The best way to understand the issue for a particular method, sample size, and data type is to perform a simulation study.
if the data are not normally distributed,check data with robust regression outlier.
"Data" can never be normal; the normality assumption does *not* refer to the observed data. Rather, the assumption is that the *process* that produces the data is a normally distributed process. And that process, likewise, can never be precisely normal, because of asymmetries, discreteness, and boundedness of the observable data. So the question is never "normal or not". It is always "how different from normal is the distribution that produces the data?" When that distribution is close to normal, the methods should be fine. And, depending on the method and sample size, the methods may be ok even when the distribution is far from normal because of the Central Limit Theorem. There is no "one size fits all" answer, rather the answer is always, "it depends." The best way to understand the issue for a particular method, sample size, and data type is to perform a simulation study.
A simulation study is a fairly good idea, and R is a great resource for conducting that sort of study. Just be very clear on the null hypotheses, as these will form the core of the effort. Also, you will use the raw data to enter into the random number generators. The validity of the entire study hinges on the accuracy of the estimates. Often sample size makes this assumption problematic.
Assumptions of normality imply distributional symmetry and kurtosis. These are hardly attainable in practice. The use of the Central limit theorem helps us circumvent such departures from basic theoritical assumptions of normality. The theorem states that if the sample size increases indefinitely the observed distribution tends towards the normal or Gaussian distribution. The solution therefore is to take as large a sample as possible because the larger the sample the closer the situation is to the Gaussian distributional situation.
That is not quite how the Central Limit Theorem works. It also has a few assumptions.
Assumption 1) Observations are independent and identically distributed.
The CLT in this context is usually making the assumption that a measured quantity is a combination of many unmeasured distributions. The simplest example is to roll a 6-sided dice. The resulting distribution is uniform (1,6). Any single rolled 6-sided die will have the same U(1,6) distribution. If I roll 25 of these dice, the CLT predicts that the resulting distribution will be close to Normal (and it is).
Assumption 2) The weight of all factors is roughly equal. If you have the combined result from 19 distributions that are uniform that are weighted to account for 5% of the outcome and one distribution that is Poisson and weighted to account for 95% of the outcome then the combined effect will be roughly Poisson.
The reliance on the CLT to save you from issues of non-normality is misguided.
But having different "weights" or different distributions contradicts your initial "iid" assumption. Anyway, CLT *does* work with different weights and even dependencies; see a good book on probability theory for the assumptions needed on weights and dependence structures.
A bigger point is that CLT applies to the distributions of sums (or averages) for different n, not the actual data distribution, as the original post seemed to imply. The data distribution never changes at all for different n, by definition, when the process is iid.
Dear You can smoth your data using 3 runing means or four runiing means for example
2 3 runing means 2+(2*4)+6\4
4 2+2*4 + 6 /4 = 4
6 4+2*6+4\4 = 5
4 6+ 4*2+5\4 = 4.75 or visit moving average
https://en.wikipedia.org/wiki/Moving_average
In statistics, a moving average (rolling average or running average) is a calculation to analyze data points by creating series of averages of different subsets of the full data set. It is also called a moving mean (MM)[1] or rolling mean and is a type of finite impulse response filter. Variations include: simple, and cumulative, or weighted forms (described below).
Given a series of numbers and a fixed subset size, the first element of the moving average is obtained by taking the average of the initial fixed subset of the number series. Then the subset is modified by "shifting forward"; that is, excluding the first number of the series and including the next number following the original subset in the series. This creates a new subset of numbers, which is averaged. This process is repeated over the entire data series. The plot line connecting all the (fixed) averages is the moving average. A moving average is a set of numbers, each of which is the average of the corresponding subset of a larger set of datum points. A moving average may also use unequal weights for each datum value in the subset to emphasize particular values in the subset.
A moving average is commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends or cycles. The threshold between short-term and long-term depends on the application, and the parameters of the moving average will be set accordingly. For example, it is often used in technical analysis of financial data, like stock prices, returns or trading volumes. It is also used in economics to examine gross domestic product, employment or other macroeconomic time series. Mathematically, a moving average is a type of convolution and so it can be viewed as an example of a low-pass filter used in signal processing. When used with non-time series data, a moving average filters higher frequency components without any specific connection to time, although typically some kind of ordering is implied. Viewed simplistically it can be regarded as smoothing the data.
In financial applications a simple moving average (SMA) is the unweighted mean of the previous n data. However, in science and engineering the mean is normally taken from an equal number of data on either side of a central value. This ensures that variations in the mean are aligned with the variations in the data rather than being shifted in time. An example of a simple equally weighted running mean for a n-day sample of closing price is the mean of the previous n days' closing prices. If those prices are p M , p M − 1 , … , p M − ( n − 1 ) {\displaystyle p_{M},p_{M-1},\dots ,p_{M-(n-1)}} p_{M},p_{M-1},\dots ,p_{M-(n-1)} then the formula is
https://en.wikipedia.org/wiki/Moving_average
No problem !!
if your data distribution is not normal or even if you are not sure, you have to use non-parametric tests for data comparison.
Normal data distribution is not an obstacle against statistical analysis.
Well, since the distribution that produces the data is *always* non-normal, then the logical conclusion would be that you *always* have to use non-parametric methods, and *never* any method based on an assumed normal distribution. Clearly, that is neither generally accepted nor useful practical advice. The over-simplistic 'logic' of "not normal => use nonparametric methods" is flawed. See my post above for a more reasoned approach.
you can increase your sample size. I agree that not all data is normal and depending on what you want to do you can adopt non-parametric test. otherwise if you carry out a parametric test you will get values yes as your output but they will be meaningless
Biological data is *always* non-normal. With a sufficient sample size you will always be able to reject the null hypothesis in tests for normality. So now you use a test that derives the underlying distribution based on the observed values. How many observations do you need to clearly distinguish between normal, uniform, gamma, beta, Poisson, negative binomial, skewed normal, or any of the several hundred other possible distributions and their variants? Generally, the appropriate sample size is larger than most research projects can support. So you have several choices. 1) Get paid for 10 replicates and do another 1000+ for free. 2) Convince a funding agency that they only need to fund your research. 3) Refuse funding because the agency will not provide adequate support. 4) Accept funding and try to figure out a method that will work with the 10 replicates that you can get. In the latter event you can use parametric or non-parametric methods. For parametric methods you assume that everything is normal (knowing that answer is wrong). For nonparametric methods you avoid the assumption of normality (knowing that you have insufficient sample size to power the nonparametric method). In the face of small sample sizes, one approach is not superior to the other. They just have a different set of evils.
The following options may be of help:
(1) Account for non-normality by addressing possible reasons such as presence of outliers, overlap of two or more processes, insufficient data discrimination, sorted data, values close to zero or natural limit, etc.
(2) Increase your sample if possible
(3) For small sample, run an equivalent non-parametric test
(4) Transform data with function that will force it to fit normal model
It is possible to remove sufficiant outliers to either make the data normally distributed or to reduce sample size below what is needed to reject the null hypothesis that the data are normally distributed. This is a perfect example of torturing the data enough to make it confess to anything.
Please do not remove outliers unless you can identify a specific cause. Please be consistent. "This value is 4 standard deviations below the mean" and after looking at field plots we discover that this value came from a block that was flooded for 2 months. In this case please delete the value and depending on the degree of flooding all other values from that block regardless of outlier status.
The gender ratio at birth seems to show similar behaviour seemingly due to the intermittent effects of a presumed infectious agent, see DOI : 10.9734/BJMMR/2017/32372 This same infectious agent also seemingly affects females more so than males, see http://www.hcaf.biz/2010/Publications_Full.pdf which seems part of wider immune differences between the genders
@Rodney, Could you please help me see how the answer is relevant to the question? If the answer is in response to an answer given for this question could you help me find that?
Try to fit the distribution to data and discover what type is it.
I also have the same problem of non normal distribution. Please suggest me what to do?
For normality you can increase your sample or remove outliers (to force it look normal). But if your data in not normal, that is fine. you need to work with that and use models/functions that don't assume normality of data distribution.
You should not simply remove points that you view as being outliers. What if some of these extreme points suggest a heavy tailed distribution? Not all statistical analyses require normailty of the response variable. In regression analysis it is sufficient to show that the residuals are roughly normal.
its also happened to me. After normality test, based on the skewness and kutosis value, it shown the data was normally distributed (greater than p-value 0.05). However, the result of kolmogrov-smornov and shapiro-wilk test shown both p-value less than 0.05. I also dont know what to do next. Is it ok if I proceed with the non-parametric test?
The nonparametric test will be fine either way. The nonparametric test does not assume normality but will still work if the data are normally distributed.
@Sharif Shofirun What is your sample size? If enough big, consider Peter H. Westfall's advice "the methods may be ok even when the distribution is far from normal because of the Central Limit Theorem".
@ Maciej Karczyński my sample size is 400. Thanks all for the respond. I really appreciate.
@Maciej Karczyński. I am working with panel dataset (N=100 and T=21).Diagnostic test reveals the presence of heteroscedasticity and Non-normality of almost all variables. Dependent variable is also highly skewed. How I can proceed with such data set?
Not sure if this thread is still active, but thought I'd give it a go...
I am working with panel data - sample is 5371. I am working with scales that are already validated and reliable. However in early explore analysis the data is not normally distributed. Can anyone suggest any functions/models or justifications to continue with this data set?
Many thanks
You can add a logarithm of the variable and check for normality, if it is still not normal then too many outliers
Deleting data values because they are outliers in some assumed distribution is a very bad approach to data analysis. This strategy ignores the likely occurrence that the assumed distribution is wrong. It assumes that a small sample size can accurately model the underlying distribution. It assumes that one will never observe an unusual value in a small sample size, thereby ignoring the probabilistic nature of random sampling.
It is possible to prove that the sample mean is an unbiased estimator of the population mean. Therefore it makes sense that it would be impossible to prove that the mean of a "sample with outliers deleted" is also an unbiased estimator of the population mean.
Lastly, please read the answer provided by Peter Westfall.
I did not understand what should I do if abnormal data were distributed for a time series Peter H Westfall
I am carrying out a multiple regression and my dependent variable fail the the normality test but other assumptions were met, should I continue? The sample size is 164 and the numbee of indep variables are 6
- Badges
- Science topic