I have a within-subjects design where I asked all respondents two kinds of questions, both with only two options (either correct or incorrect). I want to know whether there is a statistically significant difference between correct responses in one type of question vs the other.
I initially thought of doing a paired t-test since the samples are related but I found that the paired t-test is only appropriate when the dependent variable is continuous. In my case, my dependent variable is binary (0 if incorrect, 1 if correct).
Can anyone suggest an alternative method? Or since I'm looking at the proportion of correct responses, does that mean it would be acceptable to use a paired t-test?
Thank you so much in advance!
Hi, Ana,
The design and hypothesis is appropriate for a paired t-test.
- You give a single sample of subjects two tests, A and B.
- Each test has multiple questions.
- The answer to each question is either correct or incorrect.
- Converting the number of correctly answered questions to the percent of questions answered correctly standardizes the measure across people for number of questions answered (if this differs). If this does NOT differ across people, then the count of correctly answered questions is the total score for A, and the total score for B.
- Your hypothesis involves whether or not the mean (percent) correct answer score for the sample are more correct for one type of question, or for the other.
Assuming the data meet the assumption-based requirements of t-test, then the result of the t-test is statistically valid with respect to comparing mean levels.
If you'd rather compare the entire score distribution between A and B--rather than only the mean, median or mode--then there is a modern exact method which has no distributional assumptions for data to meet.
This 1-page paper explains how to use the modern method to identify the most accurate possible model for one's sample.
- https://odajournal.com/2014/04/06/a-statistical-guide-for-the-ethically-perplexed-chapter-4-panter-sterba-handbook-of-ethics-in-quantitative-methodology-routledge-2011-clarifying-disorientation-regarding/
And, these two brief papers explain the modern method.
- https://odajournal.com/2017/04/18/what-is-optimal-data-analysis/
- https://odajournal.com/2020/06/28/what-is-novometric-data-analysis/
@Ette Etuk
Do you believe (can you explain why) these data meet the requirement in logistic regression--that observations in the two groups being compared (here, scores on tests A and B obtained by one group of subjects) must be independent of each other?
I am very agree with Professor Paul Yarnold's both comprehensive replays, however., it seems to me that Ana is the very beginner in the subject she have a query.
Muhammad Ali
Thank you, Dear Sir.
In my opinion, the "very beginner" is the very best person to hear the truth--that the old methods don't work, and that new, much better methods are available.
Everyone that I know who has learned the new methods has told me that the elegance and simplicity of the modern method is MUCH easier to understand, to use, and to interpret than any obsolete legacy method.
In fact, here is a short article which was written on this very subject:
- https://odajournal.com/2013/09/19/the-loyola-experience-1993-2009-optimal-data-analysis-in-the-department-of-psychology/
- Badges
- Science topic
- Similar topics
- Correction