We are working on a study where we compare pre and post-workshop answers from a group of subjects. Subjects answered a question before the workshop and answered the same question after the workshop. The question is multiple choice, and the answer by each subject is either correct or incorrect, which we represent in our data sheet with a zero or one accordingly (binary). Shapiro-Wilk and a few other tests show that none of the variables are normally distributed, most likely due to the small sample size (N=15). What type of tests (using SPSS) would you recommend to determine if the workshop made any difference in subjects' answers for the binary nonparametric question?
Based on your description of the data and the problem you are solving, the recommendation is a McNemar test. It is intended to determine the difference between two dependent samples (before and after answers) when the answers are of the binary type.
Just to add to Zeljko's answer. Since you have such a small sample, you'll probably want to conduct the exact version of McNemar's test (odds ratio) rather than chi-square.
Since your DV is binary it appears that you have a simple binary logistic regression with a simple repeated measure structure on the RHS. What exactly is the research question... I believe more information is needed to be of assistance here. Best, David Booth
I will given different advice from Zeljko Krneta , Blaine Tomkins , and David Eugene Booth , though all of their suggestions could be done, and are probably more what you are after than my suggestion, but here goes.
I wouldn't recommend any statistical test. Here are two reasons (probably more that others can add). The first is that if you are asking the same question before and after some intervention, depending on what the intervention is any reasonably person would expect the accuracy to go up, so why test the null hypothesis. It is not a "bold conjecture." You might be interested in the size of the increase in accuracy. Second, the sample size is small for answering a single question (I'm in education, and a single item test is less reliable than a 20 item test), and see Gelman and Carlin (http://www.stat.columbia.edu/~gelman/research/published/retropower20.pdf) on Type S and Type M (sign of the effect wrong, magnitude wrong, respectively). THese are more problematic for small samples. If the null hypothesis was reasonable, the probability of a significant p being a Type S error or a Type M error are high. While n=15 is fine for a lot of research, from your description a single item test seems tricky. But, if you explain your research context more iI could be convinced otherwise.
An issue with logistic regression (logit(pr(post)) ~ pre) is that the questioner is not focusing on the association, the slope (the log odds ratio depending on the coding), but on the intercept, so you need to be careful with the coding.
Daniel Wright I don't quite agree with you. You characterization of the model is quite like what I had in mind, however if we use your characterization of the model, you say, "The questioner is not focusing on the association,..., but on the intercept" I realize that there are multiple ways to characterize an ANCOVA model , one of which is yours. But I disagree with your conclusion. The Questioner says that he wants to know the post-pre difference. This is exactly what my model and your model predict/explain though you could have either post-pre=RHS or post =pre +RHS, notice RHS here is the RHS of the post-pre model. I believe predicting that difference directly is the best way to go. HE IS NOT FOCUSING ON THE INTERCEPT he is focused on modeling the difference that he wants to know. THis is exactly ANCOVA for a binary DV and that is how I have modeled it. I do see that your approach is equivalent to the Post=Pre +RHS formulation but clearly in this case one is bettered by predicting/explaining the difference which is what is wanted rather than trying to subtract two logit values and then try to transform back to what the individual wants. In an ordinary post-pre design without logit you might have a point. In the logistic case I wouldn't wish to mess with the difference of two logits when I can calculate the desired quantity directly. Best wishes, David Booth
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size