My reading of Angoff's method is that the judges must know the responses, as without the responses they will be unable to determine the knowledge level of the minimally competent learner. It also appears the judges must have significant inter-rater reliability. As Angoff's earliest paper from the 1970s is somewhat difficult to obtain, I am counting on some secondary sources for this information, and did not see the level of correlation required for inter-rater reliability.
I think you will find this article helpful, it is available free online:
Brandon, Paul R. "Conclusions about frequently studied modified Angoff standard-setting topics." Applied Measurement in Education 17.1 (2004): 59-88.
Cheers!
Rich
Rich,
Thank you so much for taking the time to answer and explain. My gut instinct was to include the responses, but I wanted to be sure before I developed the Angoff rating form for the SMEs. I will take a look at the article that you suggested. Again, thanks so much for your time.
Sincerely,
Beverly
- Badges
- Science topic
- Similar topics
- Geoscience
- Asia
- India