I see many papers that perform statistical analysis on small numbers of participants (a few dozen or less) and claim that the findings are generalizable.
My statistical coursework years ago tells me that this is too few, but what do YOU think about the threshold of number of participants or sample size needed in order to be generalizable?
If you are coding everyone in a small class, rather than a sample of a large population, is that generalizable? I am not sure that it is, but I wonder what YOU think.
It depends what you mean by fully generalizable and what your claim is. If my claim is that "all left handed people are over six feet tall" and I find one who is not, then that one is enough.
"I see many papers that perform statistical analysis on small numbers of participants (a few dozen or less) and claim that the findings are generalizable." a good study will speak more on robustness of the statistic obtained. look more into robust measures.
How many participants do YOU think are needed for results to be fully generalizable? based on the study, this can be answered with a probability statement. WHAT SAMPLE SIZE WILL MAKE ME (e.g. 99%) CONFIDENT OF MY RESULTS WITH the barest minimum MARGIN OF ERROR. (e.g.0.01%)
Personally, i think generalization of statistical results can be argued from many dimensions. Obvious sample size tends to be the first point of call. Logically, a large sample size implies low variable with good information for good generalization. There are many shortfalls to this argument e.g the case of rare sample!, cost of increasing sample size, time constraints for the study you are on. Well this situation will not allow you to increase sample size to achieve generalization. But careful simulation constructed on good information and prior knowledge can help.
I would advise you seek a robust measuring technique which can support the findings you have. In many cases i will prefer to ensure robust measures are attained to allow for a fair deal of generalization.
Bear in mind you study criteria and constraints can even limit your ability to generalize. it then befalls on you to prove that your findings are robust enough to allow for generalization.
It depends what you mean by fully generalizable and what your claim is. If my claim is that "all left handed people are over six feet tall" and I find one who is not, then that one is enough.
So Michael, what is your definition of 'generalizability'?
John,
I define "generalizability" as results from one study group being applicable across all similar populations.
From a research perspective, it means that the results are reproducible in other groups from like populations, such as college EFL students in Asia, etc.
From a practitioner perspective, it means that the same methodology should achieve the same results in other groups from like populations, and that success in one place should predict success in another.
I actually think that many studies that claim generalizability are really not because there are too many variables that cannot be controlled.
Thanks Michael, great response. Can you see a case for generalising from qualitative findings?
Not in the sense of a perfectly random sample predicting the full population, such as is done with opinion polls, television ratings, etc.
Qualitative methodology is intended to understand the perceptions and experiences of the participants. Saying that similar people in similar contests may experience similar things is not the same as statistical generalizability.
The area in which I usually write has to do with using technology to assist language learning. Qualitative methodology is valuable here in order to understand the experiences of the students with the technology. It can elevate both problems and benefits that a survey might not, because qualitative methodology allows discussion of small details.
In this field, understanding how the technology affects learning, i.e. the affordances, is considerably more important than whether there was a statistically significant difference between people who used the technology and those who did not.
Unless the technology is designed terribly, it will almost always benefit learning, compared to not using technology at all.
To get a feel for why statistical analysis is unreliable, read the attached paper presenting language learning as a complex dynamic system with endless variables that researchers cannot control for.
Article Environmental factors affecting Computer Assisted Language L...
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis