Edited: 4/22/21 - Updated description and a more precise question.
Setup:
One needs to analyze the effectiveness of a learning game compared to a traditional lesson. For that, one collected data from two groups (game, traditional lesson). Each of them did the same pre-test, post-test and had identical learning material (exercises, theories,...) through the experiment. Only the context was different (game / traditional lesson). There were no data collected in the pre- / post-test to map the results to the individual student. So the data are in random order. The collected data is mostly non-normal distribution.
Question:
How can one examine the effectiveness of the game against the traditional lesson? In other words, how can the following statement be validated: “A lesson that consists of a game can lead to the same effectiveness as one hold by a teacher.”
Thank you in advance for your help
You can conduct a mixed ANOVA with two independent factors: context (between-subject) and time (within-subject).
A simpler alernative here is to use the (post-pre) differences of each experiment and compare these differences by a two-sample t-test.
Blaine Tomkins By going with the mixed ANOVA approach. Does it make sense to perform a paired t-test to validate the significance of the scores of each group, followed by a mixed ANOVA to examine the effectiveness between the two groups? Or is the t-test in advance a no-brainer? PS: I forgot to mention the that was anonymously collected if this does matter.
Jochen Wilhelm Thank you for your answer it sounds smart. I googled about the procedure and I forgot to mention that the pre-post was collected anonymously (sorry), so I'm not able to calculate the difference of each proband (as I googled they are required to perform the two-sample t-test). I think this viewpoint changes the approach: what would it look like then?
Kai Krause You have a 2 x 2 factorial design, and a mixed ANOVA will enable you to compare both main effects and the interaction in a single test. A t-test prior to the ANOVA is unnecessary. I'm not sure what you mean by responses were "anonymously collected". If you're saying that you can't match an individual's pretest score with the same individual's posttest score that is a problem. You won't be able to properly test the effect of the intervention in that case.
If you have only the means and SEMs of the differences (and not the individual differences themselves), then you can do a t-test on these summaries:
if d1 is the mean difference from experiment 1 with a SEM of s1 on n1 degrees of freedom, and d2, s2, n2 are the corresponding statistics from the second expriment, then the mean difference of differences is d = d2-d1, the SEM of d is
s = sqrt(s1²+s2²) and the degrees of freedom are n=(n1+n2-2). So you can calculate t = d/SEM and get the p-value from a t-distribution with n degrees of freedom.
Jochen Wilhelm I'm a bit confused. How does one obtain the SEMs of the differences without first knowing the individual difference scores?
Blaine Tomkins "If you're saying that you can't match an individual's pretest score with the same individual's posttest score that is a problem". Sadly this is exactly the case. I apologise for misleading you. I edited my original question. To make things more clear:
I try to examine the effectiveness of the game against the traditional lesson. In other words, how can the following statement be validated: “A lesson that consists of a game can lead to the same effectiveness as one hold by a teacher.”
In that case, I second Jochen Wilhelm 's suggestion above to examine the effect of the lesson modality. I'd also suggest an additional t-test on the pretest (baseline) scores between the two samples to ensure the groups were comparable prior to the intervention.
Blaine Tomkins, I feel honored that you second my idea, but I'd like to point out a flaw in your last post: a t-test on the baseline scores is not informative. First of all, the test tests if the data are sufficient to distinguish the sample estimate from a hypothesized value (e.g. "no difference"). Failing to reject does NOT mean that there are no differences or that the differences are negligible.
Further, if the individuals were randomized, then they are randomized. That is, finding a difference in baseline is a known false-positive. If they are not randomized, then the experimental design is flawed anyway and this wouldn't be rescued by testing the baseline differences. If you don't have any idea what could be the cause of the baseline differences, then it doesn't help much to find that the data are sufficient to show that; and if you know the factors (other than the experimental factor) that may play a role and that you werent able to control experimentally, then you should consider them as covariables in the model.
Testing differences in baseline values something very typical in the "Table 1"`s of medical publications. This is usually ... I don't now how to phase it gently ... "stupid" and should not be taken as "good scientific practice".
Jochen Wilhelm I'm not suggesting that a null result means there are no differences among the groups. There is most certainly some between-group variability. My concern is the following: Suppose that Group A's pretest scores (prior to training) are significantly worse than Group B's. This is something that one is not likely to anticipate - particularly if the researcher used random assignment. Furthermore, you likely wouldn't know the cause of it unless it was something particularly salient. It could be due to any number of variables, and even possibly due to error.
Say we decide not to compare the groups at baseline, and only conduct a test on the pre/post differences between the two groups and find that Group A's improvement was significantly larger than Group B's - unbeknownst to us, Group A had less knowledge at baseline than Group B (as indicated by a lower average score). Knowing that the groups differed in terms of knowledge at pretest would change the way we interpret the results. Also, in the particular case, we couldn't include pretest scores as a covariate - at least, I would be reluctant to.
Blaine Tomkins , I fully agree: it makes sense to cross-check that the data look sensible. But you don't get to that with hypothesis tests. Statistical significance is not equal to practical significance (or relevance). Non-significant results may simply miss to indicate potentially relevant differences, and significant results may just show that the data are sufficient to discern totally irrelevant differences from zero.
Your example mixes the view on a particular case with the statistical view. If the data were randomized and the experiment was performed correctly, then the test will have the desired properties. The possibility that the randomization accidentally lead to baseline differences is "priced in". This refers to *all* the properties of the subjects, not only to those very few you might be aware of and able to observe or measure!
Jochen Wilhelm Interesting points. Now I'm curious how to respond to criticisms of the type "how do you know the samples were similar at pretest?" that are still very common in my field - even when assignment is random!
What you're saying is - if assignment was random and the experiment was conducted correctly this is, ultimately, a ridiculous question. Correct me if I'm wrong.
Two points.
1. You haven't said how people were allocated into the groups (e.g., influenced by the pre-test?, by larger units like classrooms so the data are not independent?, etc.). Without this information it is difficult to address the effectiveness question.
2. It sounds like the learning materials was composed of several items, and these were the same for the two conditions. Therefore it is likely that you will want to take this into account.
- Badges
- Science topic