I am trying to understand the population's knowledge on the specific topic before and after the manipulation. Since the participants are anonymous I can't use paired sample t-test. But I was wondering if I could a- calculate the mean of the first test and second test and use paired sample t-test between the first and the second tests; b- pairing the questions and try to figure out the success rate of each question (I don't know how to do that though).
N=40.
I'm assuming that, for each student, you have several questions, that are either right or wrong, and that for each student these are compiled into a single score, e.g. 75% correct. Is this right?
If this is the case, you can compare the mean of before scores to the mean of after scores with an independent samples t-test, or another two-sample test, as appropriate, like Wilcoxon-Mann-Whitney, Mood's median, and so on. However, this will violate the assumption of these tests. But there's not much you can do about that if the responses are anonymous. The only thing you can do is explain this to your audience.
You can also compare the results for each question. The same caveat will apply about violating the test assumptions of independent observations. You could use a chi-square test of association, (or independence), for a 2 x 2 contingency table for each question.
Hi Ömer & Salvatore, I know it feels like you should be able to adjust your t-test to within-subject (e.g., change degrees of freedom). But actually you can't since the more your pairs are correlated, the higher your t-ratio and lower your p-value for the same data. Here's an example I made up with a mean of 8.50 (sd 3.03) at time 1 and a mean of 11.00 (sd 6.06) at time 2. I literally just shuffled the numbers for T2! Good luck with your study! ~ Katie
---------------------------------
T1 T2
4 2
5 4
6 6
7 8
8 10
9 12
10 14
11 16
12 18
13 20
t(9)=5.7446, p=.0003
---------------------------------
T1 T2
4 2
5 18
6 6
7 14
8 10
9 12
10 8
11 16
12 4
13 20
t(9)=1.3374, p=.2139
---------------------------------
You can use only knowledge that you have. You don't have the knowledge about which values are paired, so you can't use it. And there is no secret trick that would provide you this knowledge. So you must treat the two samples as independent. By doing this, you double the degrees of freedom at the cost of the between-subject variance that is not explained (and limits the precision of the mean-difference estimate). Depending on the between-subject variation this can be a disadvantage (unexplained variation wins over degrees of freedom) or an advantage (degrees of freedom win over unexplained variation).
I would not be as strict as Sal Mangiafico stating that the unpaired analysis would violate the assumptions. The analysis should use all available knowledge. I don't see a problem in using an unpaired analysis when the knowledge of the pairing structure is nor available. The only effect is that the between-subject bariance will not be explained and will just be projected into the precision of the estimate. This might be unfortunate, but not wrong.
Consider you did not ask for the gender of the subjects. This is not part of your analysis in any way. Consider further, that in two years someone shws that gender plays an important role on how your treatment affects the subject's knowledge on that specific topic. This would raise a flag that when planning a next study one should explicitely model the impact of gender (i.e. to include the gender:treatment interaction), but you didn't know this, and this does not render your analysis wrong. Instead of gender you may think of some factor that might be less obvious and that would never be actually analyzed, or of a factor that cannot be analyzed, because it's impossible to get that information (e.g. relevant early-childhood experience that isn't remembered).
- Badges
- Science topic
- Similar topics
- Statistics
- Survey