I have a set of data with n=5 in two groups. The groups are unpaired. The distribution of data in the population is unknown so, I was thinking of using a non-parametric test. I am only looking at one variable but there are many variables at play to confound the data because these are samples from human patients. I would like to be able to have a power of 0.8-0.9 and be able to detect a small effect size. I do not know what effect size I am expecting. Typically, people seem to use a student's t-test for these types of samples, but I don't think this is the best test because it is unknown as to whether the data is normally distributed due to it's small sample size and lack of information regarding the population. Any thoughts are appreciated!
It depends on what are you going to do with your data? What is your research objective? According to that you can find various tests (parametric and non parametric) that you could apply. Also it is true that you should use NON PARAMETRIC test since you have a small data set and you have no clue about the distribution of the data set. (conclusion of the results of a parametric test depends on the distribution of the data set)
There are many issues in your question.
1. What is your statistical hypothesis, precisely? This will determine the test.
2. These things don't go together: "I would like to be able to have a power of 0.8-0.9" and: "I do not know what effect size I am expecting" (or: what precisely *is* a small effect size that you like to detect) and "The distribution of data in the population is unknown". .- You need to know quite a lot to specify a power, including the precise statistical hypothesis.
Since you have so little information I really wonder what a test will help you. It looks like an exploratory study, so you should better describe what you find.
And a sample size of 5 for human data is really little (at least if you have no further information and idea about the data and the effects and all that).
Most people use a t-test in such cases because
1) they don't know anything else
2) others did this, too
3) they were told to do so
All really bad reasons.
Good that you critically think about this. I think that *not testing* is the right way. But I also think that the typicall boss / coauthor / reviewer won't be satisfied with this. So at the very end it will be that you will have to do what your boss / coauthors / reviewers tell you to do. Most likely they will call for a t-test, because of the 3 arguments I wrote above. some may tell you to do some non-parametric test like Mann-Whitney (MW) or so, because they think that this is a must if you actually want to do a t-test but you have no information about the distribution. (*)
Good luck.
(*) That is nonsense, imho. Assumptions must be justified by good arguments and by a reasonable theory - not by data. If data strongly(!) contradicts the resonability of an assumption, then we learned something new. This is in itself valuable, but requires enough data. The argument that for instance the MW-test is an alternative to the t-test is also wrong (statistically). These tests are testing DIFFERENT hypotheses. It is silly to chose a hypothesis by the test I am able to use. The hypothesis should be derived from the research question, and the test is to be selected according to the kind of hypothesis that is to be tested. Not the other way around. Yes, and it may happen that one has to acknowledge that the available data is not sufficient to answer the research question.
I have the impression that tests are chosen by people in a kind they chose a meal from a cookbook, depending on the incerdients thay have available. This is what I would call "cooking by accident". Research that should be justified by the result of hypothesis tests (whether or not this is seen as a sensible approach) should be planned, and this means that the meal is selected (or given by order) and the incredients are shopped according to the desired meal.
Just as an example: *If* your objective is to test a shift in the expectation but the assumptions required by a t-test are unreasonable, then the only option is a bootstrap-approach. However, with only n=5 there is not much bootstrapping will reveal what is not visible already with the naked eye...
I would also like to know what exactly is your research objective, and if your variable is continuous, discrete, or Likert scale. In your case though, I just don't think you have enough data to run any statistical tests since (i) you only have 5 samples, and (ii) you have other variables which means you most probably have 1 observation from 1 sub-population within each group.
Statistical significance is an interactive function of effect size and sample size. Therefore, if you are looking for a small effect you must have a large sample, and if you have a small sample you can only look for large effects.
Sure, you can always use a non-parametric test (such as the Mann-Whitney or rank-order correlation). But with a total N = 10, your ability to "find" anything in the data is markedly constrained. You are better off thinking of this as a pilot sample that may suggest an avenue for further investigation.
I completely agree with Stephen, would just like to add that if you do run a test (eg. Mann-Whitney) and result is significant with a small sample size, then it is an indication that the result is really significant.
Julie -
Confidence intervals are more practically interpretable than looking at a p-value, though you had the right idea to consider power. You might want to look at a confidence interval about the difference in the two estimated means, if that addresses any of your issues.
You may have to fall back on the Chebyshev inequality since you are not able to rely on distributional assumptions. You do not have much information, but this will be a start at exploring where you are with your project thus far. As Jochen and others noted or indicated above, planning sample designs ahead is important, and you may now have more information to plan a follow-on study. (See Stephen, re "pilot study.")
Cheers - Jim
Article Practical Interpretation of Hypothesis Tests - letter to the...
@AK Sigh:
I just read your response now, and I am curious what you think is the difference between "significant" and "really significant"?
If there were repreated measure taken on two treatment levels: control and an applied treatment...then a One way Anova with Repeated Measures followed by Holm-Sidak may work for you.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis