If an experiment includes, say, two groups and Condition 1 is identical between groups but Condition 2 differs between groups, what is the best course of action if one finds a significant difference in Condition 1 (e.g., increase sample sizes in subsequent experiments)? In other words, what is the most appropriate way to guard against this?
I'm generally interested in people's thoughts to this but it would be most relevant when considering the dependent measure to be the rate of a behavior compared repeatedly across time (e.g., sessions, time bins) in both conditions for both groups. Please see attached figure for an example of how this might look.
I appreciate your suggestions.
According to your screen shot, you seem to have an interaction effect where the impact of your independent variable is different between Conditions 1 and 2. This is an interesting kind of result, and I so I wonder why you would hope to remove it with a larger sample size?
How were people allocated to the groups? Also, what is your research question?
Thanks for your responses.
We randomly allocated participants between groups (20 per group). We're using MTurk basically to replicate some relatively simple behavioral studies.
Both groups received the same reinforcement schedule in Condition 1 but contextual stimuli changed in Condition 2 for one group (AB) but not the other (AA) -- both groups experienced extinction (removed reinforcement) during Condition 2.
The primary research question was whether responding in Group AB would decrease faster than group AA.
The difference in response rate in Condition 1 makes the comparison in Condition 2 a little more difficult and just really shouldn't be there given the identical conditions. It also makes me nervous about being confident in positive effects when they should be there.
Thanks again and please let me know if you have further questions.
Hello Christopher,
While randomization is essential for helping build confidence against a priori bias, it doesn't guarantee equality between groups (as your data clearly indicates). This is the situation for which analysis of covariance is well-suited, assuming you have a baseline measure (correlated with the DV) that you're able to use.
Using a larger sample size is one possibility, in that a priori differences of the apparent magnitude you're seeing would be less likely with larger Ns. Again, though, not impossible.
There, too, is also the possibility that some other feature (of which you may be unaware) is accounting for the differential behavior under the common treatment phase of your study. Perhaps follow-up questions with some of the participants might help determine whether you could rule this out as an explanation. Finally, in this vein, Mturk doesn't always yield the most motivated, attentive, or representative participants.
Any growth model would seem to show that your AA batch had a sharper reduction in phase 2 than the AB batch (which will certainly make it harder to support your research question).
Good luck with your work.
Thanks for your comments David.
It seems that going with larger group sizes is the most straightforward approach. We have a post-experiment survey but without already knowing what's contributing to it, it's hard to pin down exactly what else to ask to figure out what's contributing. With so little control over who the participants are and their environment, I think we just have to accept the high levels of variability in the system and just keep on trying to refine our procedures to enhance experimental control.
FYI, we're using mixed-effects modeling to analyze these data. Thanks again.
Perhaps I am dense but if you know What the contributing variables are why do you need to do another experiment?. Second, if you "go with a larger group size", Based on your comment how would you choose a larger group size and would you have to repeat the entire experiment not just "take a few more points". I am partial to Prof. Morse's suggestion of analysis of covariance. Further, since the plot seems to be "made up" data i.e. you say "how it might look" i suggest it is of little value in planning. As is sometimes said, proper prior planning prevents poor performance. BTW approriate sample size is part of planning. D Booth
Interesting. So, the random allocation from MTurk produced an odd outcome, but not so unlikely to question that whether whatever pseudo-random process it uses is problematic. So, the question then is how to account for the baseline differences. The Davids ( David Eugene Booth , David Morse ) say ANCOVA, and they are both wise, and that may be a good choice. Subtracting the baseline, taking the ratio, etc., are also possibilities, and for proper inference it depends on assumptions about how the groups would have performed in the other's condition (i.e., the Neyman-Rubin framework). So looking at similarity between your plot and the one in Wainer (Figure 2, Panel A from 1991, Article Wainer H. Adjusting for differential base rates: Lord's para...
), it seems worth re-producing his conclusion:"The very nature of untestable assumptions means that there
is no statistical procedure that can be counted on to settle this
issue. The answer to this question must come from other
sources. In this instance, however, we can be guided by some
very strong intuition. Although we can never know for sure
what each animal's heart rate would have been had we not intervened,
it is not unreasonable to believe that it would have continued
after time = 0 at about the same rate as it had before. Thus the untestable assumption underlying Method 1 (subtracting
the baseline) seems like a good bet. Thus the causal interpretation
based on the kind of covariance analysis described
here would lead us astray."
As Wainer says, this is untestable, and you research is well outside of my area of expertise, so I am just comparing plots. But I don't want you to be led "astray", using Wainer's word, but I encourage you to think about these when deciding your analytic choice. .
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size