Let’s say that I have one sample (of size n) of random variables. I want to perform one-sample t-test for following hypothesis:
- H0: population mean mu = 0;
- HA: population mean mu > 0 (one-tailed test);
According to the Central Limit Theorem, I know that my sample mean Xbar comes from an *approximately* normal distribution with the mean mu (population mean) and STD = SE (standard error). I don’t know the population standard deviation, so I use a t-distribution instead and SE = s/sqrt(n), where s is the st.dev of a sample.
Then, I calculate a t statistic as (Xbar - munull hypothesis) / SE, which corresponds to the t score on that t-distribution. And my p-value would be the probability of obtaining the results >= t score given the null value mu (0 for my example).
I have seen people doing different stuff:
- Run a normality test (like Shapiro–Wilk test) on the sample. If the test doesn’t support the hypothesis that the sample is normally distributed, then run a non-parametric test for the sample (like Wilcoxon Signed Rank test). That doesn’t make much sense, since I don’t think we care that much about the sample distribution (nor population distribution, since CLT applies for any(?) pop.distribution);
- Check if n>30, then run a t-test, assuming that normality assumption holds. This does make some sense, but 30 looks like an arbitrary number;
- Check that the sample distribution is symmetric (no extreme values) and if it is the case, then run a t-test. If not, then non-parametric test). This also makes some sense. But the question here, if I have outliers in the data, can I simply exclude them from the analysis?
So the question is, do I really need to check the distribution of a sample. And if so, why would I do that?
(included an image just to illustrate my words)
All tests (also non-parametric tests) depend on assumptions. Assumptions are idealized mathematical concepts under wich the analysis is possible. These assumptions should be first of all reasonable. If these assumptions are "exactly true" (what happens only in very, very limited and special cases in reality), then the analysis is "exact". Otherwise, the analysis is "approximate".** If an approximation is good enough needs being judged by the researcher.
Assumptions are needed not only about the distribution od the data but also about the model itself. It is assumed that the data are not purposefully collected and that they contain all information relevant to the problem. It is further assumed that the data are stochastically independent or that the correlation structure is correctly specified. Note that none of these assumptions can be tested. The researcher must take care that these assumptions are reasonable (by an adequate design of the study, including randomization, blinding, controls etc.). In the rest of my answer I will talk only about the distributional assumption, without repeatedly mentioning that it's about the distribution. It should be understood that this in only one of many assumptions.
You are correct: the approximation (and its violation) is relevant only on the stage of the test statistic, and this approximation gets better with increasing sample size. A somewhat okayish approximation of the data distribution with the assumed distribution model can result in a very good approximation of the test statistic with the derived model of that statistic when the sample size is sufficiently large. With really large samples, assumptions usually never play a role, even if the data is considerably different to the assumed model. But with such large samples, testing usually makes no sense. Instead, estimation becomes more relevant, and the precision with large samples is usually high. Assumptions are important for small samples. Only here do they pay value by increasing the power of the analysis. Ignoring a reasonable assumption is to unreasonably throw away power.
Checking assumptions is nothing bad, but doing formal tests on assumptions is nonsense.* For instance, you variable may be some concentration and you think it's reasonable to assume that the variable follows a normal distribution. A graphical check of your data (or better: of the residuals of the fitted model with so-called residual diagnositc plots) may scream at you that the values are clearly right-skewed and that the variance increases with the mean. These are loud signs trying to tell you that you should better assume a log-normal (or a gamma) distribution instead (what might have been clear from the beginning if you had been aware that concentrations must be strictly positive). If there is no clear indication from the observed data that your assumptions may not be ideal, all you have is the plausibility of the arguments that made you come up with these assumptions.
The more severe problem is that many researchers are not aware of the properties of the variable they want to analyze and that they don't think what distributional models would be reasonable. Variables with a strict positive domain like concentration was only one example. People may have binary or binomial data, or counts, or percentages or other variables on scales that are limited at either side (like typically Likert data from psychologists). There are reasonable assumptions about the distributions of all such variables (binomial, beta, beta-binomial, Poisson, negative-binomial), and modern computers and software can easily handle this.
Non-parametric models may seem a simple solution whant one is unsure about a reasonable distribution model or if the approximation may be sufficiently good. However, non-parametric models come with their own problems, a low power with small samples is only one point I already mentioned. Another point is that the interpretation of the tested hypothesis depends on the actual distribution of the data. For instance, I can construct two sets of data, A and B, for which both t-test and Wilcoxon test will give p mean(B) and median(A) < median(B). What will you conclude? Is A larger than B or the other way around? A further problem is that non-parametric tests are limited to very simple cases. You can not analyze interactions, you cannot adjust for covariables, and you cannot account of correlation structures (partial pooling or non-independence). Some of these limittations can be overcome with bootstrapping methods, which are also non-parametric. However, these methods require a reasonable sample size that is typically large enough to render the violations of the assumptions irrelevant.
----
* The assumption is idealized, and real-world data will never fit any assumption exactly. A formal test is thus always only a method to find out if the sample size is sufficient to statistically recognize the difference between idealized model and real data. Thus, having a "significant" result is by itself no indication that the observed difference is of any relevance. And if the test is "non-significant", the result means that the sample size is too small. It does NOT mean that the assumption is met, or that the differences are irrelevant (this conclusion would be based on the interpretation of a non-significant result, what is a clear no-go!). So in either way the test does not give you any useful information.
** Even if assumptions are met exactly, a test can still be approximate when approximate models are used to calculate the p-values.
All tests (also non-parametric tests) depend on assumptions. Assumptions are idealized mathematical concepts under wich the analysis is possible. These assumptions should be first of all reasonable. If these assumptions are "exactly true" (what happens only in very, very limited and special cases in reality), then the analysis is "exact". Otherwise, the analysis is "approximate".** If an approximation is good enough needs being judged by the researcher.
Assumptions are needed not only about the distribution od the data but also about the model itself. It is assumed that the data are not purposefully collected and that they contain all information relevant to the problem. It is further assumed that the data are stochastically independent or that the correlation structure is correctly specified. Note that none of these assumptions can be tested. The researcher must take care that these assumptions are reasonable (by an adequate design of the study, including randomization, blinding, controls etc.). In the rest of my answer I will talk only about the distributional assumption, without repeatedly mentioning that it's about the distribution. It should be understood that this in only one of many assumptions.
You are correct: the approximation (and its violation) is relevant only on the stage of the test statistic, and this approximation gets better with increasing sample size. A somewhat okayish approximation of the data distribution with the assumed distribution model can result in a very good approximation of the test statistic with the derived model of that statistic when the sample size is sufficiently large. With really large samples, assumptions usually never play a role, even if the data is considerably different to the assumed model. But with such large samples, testing usually makes no sense. Instead, estimation becomes more relevant, and the precision with large samples is usually high. Assumptions are important for small samples. Only here do they pay value by increasing the power of the analysis. Ignoring a reasonable assumption is to unreasonably throw away power.
Checking assumptions is nothing bad, but doing formal tests on assumptions is nonsense.* For instance, you variable may be some concentration and you think it's reasonable to assume that the variable follows a normal distribution. A graphical check of your data (or better: of the residuals of the fitted model with so-called residual diagnositc plots) may scream at you that the values are clearly right-skewed and that the variance increases with the mean. These are loud signs trying to tell you that you should better assume a log-normal (or a gamma) distribution instead (what might have been clear from the beginning if you had been aware that concentrations must be strictly positive). If there is no clear indication from the observed data that your assumptions may not be ideal, all you have is the plausibility of the arguments that made you come up with these assumptions.
The more severe problem is that many researchers are not aware of the properties of the variable they want to analyze and that they don't think what distributional models would be reasonable. Variables with a strict positive domain like concentration was only one example. People may have binary or binomial data, or counts, or percentages or other variables on scales that are limited at either side (like typically Likert data from psychologists). There are reasonable assumptions about the distributions of all such variables (binomial, beta, beta-binomial, Poisson, negative-binomial), and modern computers and software can easily handle this.
Non-parametric models may seem a simple solution whant one is unsure about a reasonable distribution model or if the approximation may be sufficiently good. However, non-parametric models come with their own problems, a low power with small samples is only one point I already mentioned. Another point is that the interpretation of the tested hypothesis depends on the actual distribution of the data. For instance, I can construct two sets of data, A and B, for which both t-test and Wilcoxon test will give p mean(B) and median(A) < median(B). What will you conclude? Is A larger than B or the other way around? A further problem is that non-parametric tests are limited to very simple cases. You can not analyze interactions, you cannot adjust for covariables, and you cannot account of correlation structures (partial pooling or non-independence). Some of these limittations can be overcome with bootstrapping methods, which are also non-parametric. However, these methods require a reasonable sample size that is typically large enough to render the violations of the assumptions irrelevant.
----
* The assumption is idealized, and real-world data will never fit any assumption exactly. A formal test is thus always only a method to find out if the sample size is sufficient to statistically recognize the difference between idealized model and real data. Thus, having a "significant" result is by itself no indication that the observed difference is of any relevance. And if the test is "non-significant", the result means that the sample size is too small. It does NOT mean that the assumption is met, or that the differences are irrelevant (this conclusion would be based on the interpretation of a non-significant result, what is a clear no-go!). So in either way the test does not give you any useful information.
** Even if assumptions are met exactly, a test can still be approximate when approximate models are used to calculate the p-values.
If the number of your observations is more than some threshold, about the mean test, you can ignore the normality presumption.
Jochen Wilhelm wrote: If these assumptions are "exactly true" (what happens only in very, very limited and special cases in reality), then the analysis is "exact". Otherwise, the analysis is "approximate".**
That summarizes very nicely what I tell my students when discussing the assumptions for the unpaired t-test. I say that if one could draw two random samples from populations that were both perfectly normal with exactly equal variances, and if each observation was completely independent of every other observation, then the unpaired t-test would be an exact test. But normal distributions do not exist in the real world, and perfect homogeneity of variance of the two populations is virtually impossible, so those conditions are never met when one works with real world data. And therefore, even though textbooks almost universally describe the t-test as an exact test, in practice it is an approximate test. And in that case, as Jochen noted, the real question is whether the approximation is good enough to be "useful" (as George Box might have said).
HTH.
I will show you by a graph.
Graph 1 is a histogram of a population (n=3858)that is not normally distributed .
Graph 2 is a histogram of a randomly selected sample (n=500).
As you see graph 2 does not appear normally distributed. The problem is, with such distribution of the sample (randomly selected, > >30) the assumption of normal distribution is not valid. Hence you cannot use parametric tests.
Tarik Qassem , you confuse sample data with sample statistic. It's the distribution of the sample statistic that is important here. And since your example data distribution is quite symmetric, already small sample sizes will give a samplemeans (= the test statistic) that with a distribution that is a close approximation to the normal model.
I simulated that in R from a distribution similar to yours, and you can seefrom the histogramsof the sample means (!) that their distribution nicely follow the normal density curve (shown in red) already for sample sizes of n=10. The histograms are based on 10000 draws of size n from the distribution shown in the first diagram.
Jochen Wilhem, i am referring to the Central Limit theorum, that states "when independent random variables are added, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed". I want ti demonstrate that the theorum does not assert that all sampleed data will be normally distributed. Hence, we can't use parametric tests without checking that their underlying assumption is valid.
Tarik Qassem , I don't get your point. Certainly, if you sample from some distribution F, the sample will also have the same distribution F (as you showed). This is trivial. As you correctly said, the CLT states that "their [samples] properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed". The sample mean is such a "properly normalized sum". The tests are about these means. They actually don't care about the distribution of the original variable at all. It's again a quite simple mathematical fact that the sum of normal distributed variables is itself normal distributed. This just means that the distribution of the sample mean is exactly normal when the distribution of the original variable is exactly normal. The CLT says that the distribution of the sample mean is approximately normal, for any arbitrary distribution F of the original variable*, and that this approximation gets better with increasing sample size and with greater similarity of F to the normal distribution.
Regarding your statement to check "that their underlying assumption is valid": If you mean "valid" in amathematical, logical sense, you are trapped in the realms of mathematics. This cannot apply to real data, since the reality is not identical to our idealized models about this reality (no real variable is exactly normal distributed, like no real round structure is a circle in the mathematical sense). So the real, actual question is never ever if or if not the assumptions are valid (they never are, in the strict logical sense), but only if the assumptions are reasonable (sufficiently well approximated or approched by the empirical reality). This is one reason why formal tests on assumptions are nonsense, as these test the observed data under the logical validity (and it's only a matter of sample size to be able to reject the hypothesis; it's based on the old misconception that a test is about the "effect" - but tests are about the data!).
*F must have a mean and a variance. If F is estimated from real observations, these conditions are always met due toe physical/practical contraints (they may not be met in theoretical constructs; The Caucy distribution for instance neither has a mean or a variance)
Ruslan Klymentiev , Tarik Qassem , Jochen Wilhelm , my two cents from the practice (putting theoretical arguments aside for a moment): the t-tests are in my experience extremely sensitive to the dispersion in the data (intuitively, because the mean is sensitive to extreme observations): the power of t tests goes low really fast as the number/size of extreme observations increases and as the number of observations decreases. When I am invited to be a reviewer of papers that include t-test I found pleased if the authors discuss or at least include the results of the tests of assumptions of t-tests, particularly homogeneity of variance and others related to the general form of the empirical distribution.
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running