Hi everyone,
As part of my PhD I'm validating a patient-reported disease severity scale for patients with a rare condition. It assesses the severity across 5 symptom groups using a 0-5 likert rating. It's been adapted from a previously validated clinician-reported version to form a lay-reported version so that patients can report their own disease severity. The symptom groups are the same but the ways in which the response options are worded are different between the two questionnaires, which means this version needs validating. Initial testing done on the questionnaire suggests the isn't much differentiation across the response options on most of the items. I was thinking about interviewing patients, amending the questionnaire and then running some quantitative analyses to validate the scale.
I'm looking at using IRT, as the scale is not to be utilized in clinical settings, as there is already a validated clinician-reported tool to measure disease severity in the population. However the main problem I face is the patient population is incredibly small and I'm unlikely to get more than say 100 participants, all the stuff I'm reading on scale development says I need a lot more data otherwise the analysis won't have sufficient power.
Has anyone got any experience validating questionnaires using small sample sizes or has any advice regarding different validation strategies?
Many thanks!
It's not obvious to me that your "sample" of around 100 would be too small. (It sounds like the cases would not be selected by probability methods, so it's not really a sample; therefore, issues like power and significance are dubious, but that issue aside ...).
You are in the rare position of having a well-validated alternative measure, the clinician instrument, to use for an examination of what's often called criterion validity. If the patient version is valid (and reliable), it ought to correlate very highly with the clinician version -- certainly, highly enough for 100 cases (or even fewer) to show the correlation as "significant." If not, then the patient version is invalid (and/or unreliable). So, will you have, or can you obtain, clinician ratings on the same patients for which you will get patient ratings?
Before testing the content domain, I will strongly suggest reading the following paper. If conceptualization is based on well-developed measurement theory, then a small sample size is sufficient for your scale validation:
(20) (PDF) Specifying the Problem of Measurement Models Misspecification in Management Sciences Literature (researchgate.net)
The creation of a scale is based on the assumption that each of the items measures the same underlying concept. I am not sure whether this matches your case of asking about different symptoms. In particular, would some with the condition in question be likely to report all five of your symptom groups?
- Badges
- Science topic
- Similar topics
- Psychology