Randomized experiments are a scientific standard method. Nevertheless, there is considerable criticism. It can be argued that scientific experiments are "artificial environments choreographed by experimental protocols" (https://errorstatistics.com/2018/03/20/j-pearl-challenging-the-hegemony-of-randomized-controlled-trials-comments-on-deaton-and-cartwright/). I tend to agree with this because experimental research designs may indeed depart from reality and thus influence results. The referenced text provides further arguments. On the other hand, the control of unwanted influences is considered a strength of the randomized experiment. Therefore, my question: What are the advantages and disadvantages of randomized experiments beyond well-known aspects?
It is possibly an element of the human condition that we want clear simple answers. We want statistics to give us those answers, but statistics is not designed to do this. We often equate "significant" with "true" and "non-significant" with false. This is an abuse of statistics. The other problem is that we often try to apply one answer to all cases. It is published so it is true.
If what you mean by "experiments are 'artificial environments choreographed by experimental protocols"' is that an experiment uses the population convenient to the experimenter, then you are correct. This is one reason why it is important to have different scientist try to replicate research results of other scientists.
You have a choice, and you cannot win. You can have a "real world" experiment wherein you cannot control many of the variables that are important. Alternatively, you can run a laboratory experiment and control most of these factors. However, your experiment will "solve" problems by holding all these "unwanted influences" constant. So you have a very precise experiment that tells you a great deal about one small slice of the real world. To get the laboratory experiment to mimic the real world you will have experiments with hundreds or thousands of treatments.
I have been looking through papers on dental experiments involving toothbrushes or mouthwashes. A typical study starts with a large number of participants. Values in the hundreds are common. Then a bunch are excluded (smoking, pregnancy, number of teeth, general health are common criteria), and the rest are broken into two to four treatment groups. At this level there may be fewer than 10 people in each treatment. Then I need to ask, how well do I fit this study group? The experiment is run to give the best possible chance of finding a treatment difference by excluding "unwanted influences" and in so doing reduces the probability that the results will apply to my case.
Given the population available to a researcher, randomization reduces the probability that the results of an experiment are biased by the researcher. This is not a guarantee that the results are unbiased even within the available population. It is simply more difficult for the researcher to be the cause of the bias. The simple example is where the researcher puts all the "good" experimental units in one treatment and the others are left as the control.
A disadvantage of a randomized experiment is that it might cause bias, though this is well known.
This is a matter between internal and external validity and less how to assign subjects. If someone does an experiment, random assignment is indispensable. Even if it causes a loss of external validity.
Harald, thank you for making that disadvantage clear. Experiments may only claim validity within their own artificial context. Nevertheless, we use them to make causal conclusions about the real world.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA