I want to implement a quality improvement protocol, which includes education, on the provision of oral care in an acute care facility. I would like to conduct a pre- and post- survey to the nursing staff about basic oral care questions, then compare the differences in survey responses after the implementation. How do I establish reliability and validity of the survey I create?
Well it depends how far you want to go. Reliability is more easily established than validity. You can start by doing simple tests like repeating the questionnaire to look at variation and by doing Cronbach's alpha (see psychometric textbooks). If you have a big data set you can run factor analysis. On validity I would start by getting a small panel of experts on quality of oral care (include patients and professionals) to consider maybe in a focus group the face validity of the questions - do they make sense. Then over time I would look at the correlation between your data and other data sources about the facility and are they moving in the same direction - use any quality measure you can get including staff assessments, complaints, clinical outcomes. It would be great to see if you can get a unit known to be good (or bad) on a variety of measures to run the survey and see if it puts the unit in the same ball park. if it doesn't it does not mean it is not measuring something of course - and this is the whole problem with "validity". Do as much as you can and use your common sense to assess the instrument would be my advice.
Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).
Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at test-retest correlation between the two sets of scores.
A second kind of reliability is internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. Inter-rater reliability is the extent to which different observers are consistent in their judgments. Interrater reliability is often assessed using Cronbach’s α when the judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they are categorical.
The validity of a survey is considered to be the degree to which it measures what it claims to measure. Several different types of validity must be considered when designing and deploying survey research instruments. Construct validity refers to the extent to which operationalizations of a construct (i.e., practical surveys developed from a theory) do actually measure what the theory says they do. Convergent validity refers to the degree to which a measure is correlated with other measures that it is theoretically predicted to correlate with. Content validity evidence involves the degree to which the content of the survey matches a content domain associated with the construct. Internal validity is an inductive estimate of the degree to which conclusions about causal relationships can be made (e.g. cause and effect), based on the measures used, the research setting, and the whole research design.
For further details you may refer to Research design: Qualitative, quantitative, and mixed methods approaches/John W. Creswell.—3rd ed. p. cm
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Probability
- Probability Theory
- Reliability