We use Cronbach alpha for non-adaptive test to check internal consistency of items measuring the same construct. What is the equivalent of this for an adaptive test?
Dear Naoko,
there are several papers dealing with the topic reliablity in CAT. One of them is
Nicewander, W. A. (1999). Some reliability estimates for computerized adaptive tests. Applied Psychological Measurement, 23, 239–247.
I suggest having a look there.
Best regards
Andreas
Coefficient omega uses information from factor analysis to generate reliability measures - meaning to say, where and how the variables fall into factors matter. But there isn't a "the measurement" so far and even alpha has its obvious shortcomings. You have a very interesting question.
Dear Naoko,
there are several papers dealing with the topic reliablity in CAT. One of them is
Nicewander, W. A. (1999). Some reliability estimates for computerized adaptive tests. Applied Psychological Measurement, 23, 239–247.
I suggest having a look there.
Best regards
Andreas
Hi Naoko, in general, remember that computer adaptive tests are based on item response theory (IRT) algorithms which are fundamentally different from classical test theory (where we get reliability estimates such as Cronbach's alpha).
In item response theory, reliability is a function of a parameter called item information which is the inverse of the standard error of measurement for each item across the entire range of the ability continuum (the theta scale in IRT). This can be aggregated into a total test information calculation as well. As Andreas mentioned above, there are competing ideas about the best way to capture this as a reliability estimate.
It is not as straightforward as Cronbach's alpha because IRT is a completely different approach to conceptualizing the way that individuals respond to items on scales and tests. The article Andreas cites is a good place to start. I would recommend looking up subsequent articles that have cited that paper (given its age) for a more contemporary reference. You may also check out a book titled "Item Response Theory for Psychology" by Embretson and Riese.
Good luck!
Brendan
Thanks so much for your extremely useful suggestions, everyone! I will check out the sources.
If the adaptive test is driven by purposive selection based on IRT parameters, then the more informative statistic is the (conditional) standard error associated with the estimation of the examinee's proficiency level--smaller is better, of course. If you happened to have all of the item difficulties and inter-item correlations or covariances from an appropriate group of examinees for items administered to a single examinee, the (implied) coefficient alpha--for that specific subset of items--could be estimated via [k/(k-1)]*[1 - sum(item variances)/total score variance], where k = items attempted and total score variance = sum of item variances + 2(sum of unique item covariances).
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS