I am interested in the evaluation of the responsiveness of psychometric instruments used in psychotherapy outcome research. Does anyone know some recent research on good way to evaluate the responsiveness of said measures?
Thank you all for your help!
Best regards,
David
You keep using the word "responsiveness" without defining it.
Psychometric instruments are thoroughly tested before being released for use, and are studied retrospectively as well. This testing shows that most, if not all instruments are highly reliable in their predictions or evaluations. I'm not sure, though, that this statement answers your question, because the question is unclear. You could, for example, be interested in how many subjects who are given an instrument complete it. Or how many answer honestly and fully. These are also valid and interesting questions.
There is quite a lot of literature in this area. The usual distinction is between responsiveness, which indicates how much scores on a measure change and sensitivity , which ties responsiveness to an external criterion. Psychotherapy researchers are not very focused on these concepts but they can have important implications, particularly in session by session monitoring where insensitive measures may fail to pick up true change. This has an implication that may appear paradoxical that detecting small change reliably requires longer measures. A rule of thumb would be to use around 20 items in a well developed measure.
Article Scientific and Clinical Considerations in Progress Monitorin...
Jeremy, Thank you for the definitions. I have used the STAI Form Y in research, and know from its documentation that it was not only tested for responsiveness, but also for sensitivity (by comparing with other independent measures, particularly as applied to the normal population).
So a good way to evaluate responsiveness is to show sensitivity, by comparing with independent psychometrics, or other kinds of independent measurements
In a hospital setting, for example, a new psychometric measure can be compared with nurse or doctor evaluations, DSM V evaluations, or MMPI results, etc.
I guess the sensitivity question is along the lines of "How good is this measure at detecting real change when it occurs?"
Any measure produces "signal" data systematically related to the construct under study and noise data (more or less random). The ratio between the two signal:noise tends to go up with the selection of appropriate items in the right format but also with questionnaire length.
The finding that increasingly brief measures tend to correlate with longer measures of the same thing has led to the use of very brief measures that are in the UK idiom "short measure".
This hasn't prevented such absurdities as many institutions using a single item "would you recommend this (service) to your friends and family" regardless of its unreliability. I would argue that the ORS and SRS although clinically useful suffer in a similar way.
In the development of my measure sPaCE I was on the point of developing briefer variants, when I thought the process through and realised that shortening the measure caused more problems than it solved.
Article The development of a brief distress measure for the evaluati...
Jeremy, Thank you so much for your interesting comments. That certainly makes sense. I can understand the motivation to use fewer questions when dealing with young, old, or ill subjects, which I have seen reflected in many questions on this forum, but I agree that it is hardly worth watering down the statistics like this for a bit of convenience.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics