I'm very much interested in other opinions on this topic, because the gold standard of random sampling is often not possible, very difficult, terribly impractical, or introduces its own confounding variables (e.g. lab-like conditions as opposed to a naturalistic classroom environment).
In my field, much, if not most research involves intact classes of students. Of course, anytime we conduct a quasi-experimental study, we must ensure that we control for between group differences, but we can only control for the known differences. Yet random sampling is so powerful because provided the N size is big enough, group differences, both known and unknown will be washed out.
What I'm wondering is, provided that cluster samples were formed in a way that reduces or minimizes biases, can we have more confidence that unknown differences between intact groups will also be washed out? Let's say that, for example, students in program x are assigned by admin to different classes a, b, and c based on something seemingly random, like a student ID number, and there was no other pattern or influence in how they ended up in a particular class. Can we then say that the intact classes (clusters) were, in a sense, already randomized (I do realize it might not qualify as random in the purest sense, as even a number, last name, or birthday could introduce something non-random about it), and therefore have some measure of confidence that unknown between-participant differences will also be washed out, or at least more washed out than they would be otherwise?
What prompted me to ask this question is I often see research papers in good journals that report intact classes, but they don't seem to mention how the intact classes were formed, (other than saying that they were the same major, proficiency, etc.--the known aspects). It seems that this could have important implications for how confident we can be about the internal validity of the design. Please correct me if I'm overlooking something here.
Thank you!
Kris
- Badges
- Science topic
- Similar topics
- Academic Journals