Hi,
I have developed a semi-structured interview assessment tool for a clinical population which gives scores - 1,2,3 (each score is qualitatively described on ordinal scale), using a mixed methodology. The tool has 31 questions.
The content validation was done in the Phase 1&2 and the tool was administered on a small sample (N=50) in Phase 3 to establish psychometric properties. The interview was re-administered on 30 individuals for retest reliability. The conceptual framework on which the tool is developed is multidimensional.
When I ran cronbach alpha for the entire interview it is .75 but for subscales it is coming between 0.4-0.5. The inter-item correlation is pretty low and some are negative and many are non-signifcant. The item-total correlation for many is below 0.2 and also has a negative correlation- based on the established criteria we will have to remove many items.
I am hitting a roadblock in analysis and was wondering if there is any other way in whcih we can establish reliablity of a qualitative tool with low sample, or other ways to interpret the data where a mixed methodology has been used (QUAL-quan).
Since the sample is small I will be unable to do factor analysis, but will be establishing convergent/divergent validity with other self-report scales.
Thanks in advance
Jeevitha Gowda R
when you use an AI to answer a question, please cite it just as you would any other external source that you used.
David L Morgan Do you have any insight on this that could help me in analysis?
Do I need item-analysis at all, considering it is a mixed methodology clinician-rated semi0structured interview.
I have to admit that I am unsure about your whole enterprise. First, why do you want to calculate reliability on a semi-structured interview? This is typically a form of qualitative data collection, so you would not expect a lot of consistency from one person to another. And, I am unclear about how you assign an ordinal score to a qualitative response.
David L Morgan We wished to develop a psychometrically sound tool that has a uniform rating system allotted to it. And hence, decided to run the usual reliability analysis. But yes agreed, it’s not meaningful to calculate reliability.
the 1,2,3 rating again was to standardise the answers since it was supposed to be scored by the clinicians. Mostly based on the behavioural anchors like frequently, mostly, always
Is there any other suggestion for psychometric evaluation ?
To get a high level of internal consistency, all of your measures need to be positively correlated. So, each of your scores would need to indicate frequently, mostly, always for strongly related behaviors. If any of them are in the "wrong" direction (i.e., negatively scored), they will be to reversed.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire