Hello, Is it generally true that effect sizes are lower for ecologically valid field experiments compared with controlled lab experiments?
I'm not sure about this and I'm just trying to understand why the effect sizes in my field experiment was much smaller than my experimental self-report study, on the exact same IVs and DVs of course. Do you know anything about this? Some sources on this would help very much.
Many thanks for your help in advance!
Good question. (Note: I won't try to interpret "ecologically valid" since it means different things to different people.)
It is not generally true (but there are not many things that are generally true), but when designing laboratory studies the researcher will often use the controlled laboratory setting to try to maximize (within limits) the effect size for the difference in whatever is being manipulated. In the field in is not always possible to maximize the difference between your experimental conditions. In non-experimental field settings, nature does not purposefully maximize the difference between conditions.
Hi, Pelin, usually the effect size or the minimal standardized difference we like to detect is given by the experimenter in adc´vance and from this the sample size is calculated to reach just this. When you do not design the experiment in advance then you have to live with the effect size coming out. In ecolocigal date the variance max be larger than in other fields so that the effec size for a fixed sample size is larger than in other fields. R-programs for all this you find in our book:
Rasch, D.; Pilz, J., Gebhardt, A. and Verdooren, R.L. (2011)
Optimal Experimental Design with R,
Boca Raton, Chapman and Hall,
Hi Pelin, when Brian Mullen was here at Kent he said something that always stuck with me: that effect sizes in messy, real-world studies can, counterintuitively, be larger than controlled studies. While the latter are better at removing extraneous and confounding variability from independent variables, they often strip those same independent variables of much of their impact. The emotional intensity or threat, for example, elicited by a laboratory manipulation (or either ethical, practical or ethical reasons) is often much less intense than the real-world phenomenon that it attempts to simulate. So to use an analogy with drug trials, it might be that the drug is less pure in the lab study, but the dose may be much higher in the field study. In psychological research (in line with research on situated cognition), the environmental context of field studies may also afford more sensitivity to the independent variable. In keeping with this idea some meta-analyses show larger effects in field studies, e.g. (I think):
Kraus, S. J. (1995). Attitudes and the prediction of behavior: A meta-analysis of the empirical literature. Personality and social psychology bulletin, 21(1), 58-75.
Here's just one of the meta-analyses that Brian published showing larger effects for more ecologically valid contexts or stimuli - not exactly field vs. lab, but the same point.
Mullen, B., & Hu, L. T. (1989). Perceptions of ingroup and outgroup variability: A meta-analytic integration. Basic and Applied Social Psychology, 10(3), 233-252.
One can find these quite easily on google scholar - perhaps it is time for a meta-meta-analysis to see whether it's a general finding across research domains or what the boundary conditions are!
Cheers!
Robbie
Hi Robbie! I saw this late. I see how the effect sizes can also be larger in a field study. Origianlly, I also thought that my effect sizes should have been stronger in my field study. But I checked the SD's and they were much larger in my field study data. So that explained it. I will keep these sources for reference. Thank you!
- Badges
- Science topic