Hi
I conducted a questionnaire consisting of 28 questions, using a likert scale (three options being: agree, disagree, and not applicable). The questionnaire was conducted using 2 cohorts: one cohort was my target group, and the other is a comparable group. Following collecting data, my supervisor suggested that the questionnaire should be validated. He also suggested that I remove 6 of the questions because in 2 questions all participants responded with "agree" and therefore he viewed the questions as constant, and 4 questions were only applicable to my target group, therefore, the comparable group all responded with "not applicable". These 6 questions are important for discussion for my target patient group, would you also agree that I remove these six questions from analysis/validation. My Cronbach's alpha (without the 6 questions) is 0.6. What is your view? Thanks
Hello Erini,
The type of data that should be collected (and resulting analysis) when trying to muster "validity evidence" depends on the intended use(s) of the scores. I wasn't clear after having read your query as to what that use might be. So, my answer might not be as specific or focused as you might prefer.
Here's a quick link to some common approaches to gathering initial evidence for the potential utility of a survey: Article Developing questionnaires for educational research: AMEE Guide No. 87
However, if your intended use is to discriminate persons in one target group from persons in another, then you would opt for items/stimuli that, as a set, showed greatest differentiation between those groups (as in discriminant analysis or logistic regression). If your intended use is to predict scores on some other measure or outcome, the best approach is to collect scores from persons on both your measure and on the intended outcome, and choose items (individually or as ensembles) that yield scores which best correspond to scores on the external criterion. Finally, if your intended use is to make inferences about one or more latent measures (factors) that motivate peoples' response to your items, then factor analysis is a likely approach for an analytic method. As well, you'd like to show that scores on the latent variables correlate well with scores on related latent variables (convergent validity evidecne) and do not correlate well with scores on unrelated latent variables (discriminant validity evidence).
Sorry if this is too wordy, but the literature on validation is huge, and it is hard to summarize in 25 words or fewer in a way that could guide an initial foray into instrument development.
As regards the issue of items that show no variation it is true that these (if the no variation condition continues to hold in the target population) will not contribute to either enhanced score reliability or enhanced criterion-related validity. They do no harm, however, except to lengthen the measure.
Good luck with your work.
Thank you David. This is a study in my thesis, the aim of which is to determine patients' (a target patient group) preference for a health care model. Therefore, I am trying to compare my target group's preference(s) Vs that of a control/comparable group. Thank you for the answer and the link.
Hi! Pre testing and Pilot testing can solve the validation issue prior the main data collection. Besides using the established scales also can reduce further problems of instrument validation as well. Thanks
If you are comparing the 2 groups, then you have to remove the 4 questions but not necessarily the remaining 2 questions, because if you remove those 2 questions you have no idea about the 2 groups are agreeing for a particular health care model. As per as your Cronbach's alpha value is concerned it is 0.6, which is very low and as a general thumb rule it should be 0.7 and above.
- Badges
- Science topic