I have the following situation: A number of texts have been rated for readability. The research question is: Is the used readability score valid? We did a study where participants first read the text and then answer a multple choice question. Because the choices were exclusive (exactly one correct choice), there is a 25% chance of guessing.
We ran a logistic regression with correct/incorrect as response and readability score as predictor. We find the desired relationship, but we also want to predict performance at texts of lower readability than in our sample. How can we formulate the model or adjust the estimates such that predicted probability of correct answer would never fall below 25% ?
Note that the data analysis is done by a bachelor student. Pragmatic solutions, using standard software, are preferred over profound. Also note that we have to use either a mixed-effects model or GEE to account for repeated measures.
Issue #1: "we also want to predict performance at texts of lower readability than in our sample." A simple reminder that extrapolation is always dangerous.
Issue #2: You have to allow the model to produce the full range out possibilities from predicting all 0s to all 1s. It's possible that under some conditions people would actually produce performance below 25% (e.g., if the subject was wired backward such that wrong answers looked more correct than right answers).
I'm guessing that the question arose because in trying to extrapolate (Issue #1), you generated predicted values below chance which isn't what you expected. Well, that's the trouble with extrapolation!
So, the real answer is not statistical - be sure that your data set includes texts of lower readability to ensure that the model fits these conditions and thus your prediction doesn't require extrapolation.
A completely different answer is to aggregate data to create a percentage score and then convert the score to a d' using signal detection theory where 25% = d' of zero. See Hacker & Ratcliff (1979) in Perception and Psychophysics. The drawback is that aggregation loses information, but I think this matches your conceptualization of the problem. It may not help the extrapolation issue, however....
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression