If i am developing a questionnaire that involves open ended questionnaire for a specific population, what are the most essential steps that i have to follow? Also, what are the statistical methods i can use for content validation?
I've just given a similar answer to another RG member.
As follows:
Validation is a long term process. It is concerned with the degree to which a questionnaire reflects reality. It is also about the questionnaire actually measuring what it is designed to measure.
It is always good to use validated questions from literature - directly with respondents or try to adapt them. Validity is linked to good questionnaire design, (such as inputs from experienced researchers, piloting with experienced researchers as well as respondents). The sampling frame and methods need to be robust, with appropriate rationale and determinations of sample size and, if necessary power.
Reliability is also an important consideration the degree to which a questionnaire will produce the same result if administered again, or the test-retest concept. It is also a measure of the degree to which a questionnaire can reflect a true change. As indicated already, there are a number of different aspects to validity. The degree to which questions within an instrument agree with each other is internal validity i.e., that a subject will respond to similar questions in a similar way. It also affects the likelihood of producing false positives and false negatives. The ability to make generalizations about a population beyond that of the sample tested is external validity. The degree to which the instrument can identify a true positive is sensitivity, for example, accurately identify a person who does have a belief. Similar to sensitivity is specificity, this is the degree to which the instrument can identify a true negative, for example correctly identifying the people who do not have the belief. Sensitivity and specificity are the flip side from internal validity. Related to internal validity is statistical validity, and assesses whether the differences in the questionnaire results between groups can appropriately be subjected to statistical tests of significance. There is also Longitudinal validity: whether a questionnaire returns the same results in a given population over time, assuming all else remains equal and in a social consideration there is Linguistic validity: whether the wording of the questionnaire is understood in the same way by everyone who completes it. The latter has implications when translating questionnaires for different languages and cultures.
Importantly there is content, construct and criterion related validity to consider.
Content measures the degree to which the test items represent the domain of the trait or property being measured. In order to establish the content validity of a measuring instrument, the researcher must identify the overall content to be represented. Items must then be randomly chosen from this content that will accurately represent the information in all areas. By using this method the researcher should obtain a group of items which is representative of the content of the trait or property that is to be measured
Construct must be investigated whenever no criterion of content is accepted as entirely adequate to define the quality to be measured, construct here explains some aspect of human behavior, such as physical ability, intelligence, or introversion.
Criterion is concerned with detecting the presence or absence of one or more criteria considered to represent traits or constructs of interest. One of the easiest ways to test for criterion-related validity is to administer the instrument to a group that is known to exhibit the trait to be measured.If the researcher has developed quality items for the instrument, a culling process should leave only those items that will consistently measure the trait or construct being studied.
Validity is also linked to Reliability (of which there a a range of types). There are Retest, Alternative Forms, Split_Halves and Internal Consistency methods. The latter finds wide use in Education and Social Science questionnaires
Hi, Good Morning,
Very careful with formng the questionnaire. The contains of the questions are very much related to your study. For finding the validation you have to use the ROC analysis in SPSS.
Is it not possible to create close ended questions ? Or options to choose an answer ? It is very difficult to get a standardised response from the participants. I am sure there will be a way to change most, if not all, parts of the survey into a closed ended format.
If you ask me an open ended question: "How much pain are you feeling in your throat after a 3 hour recital / performance ?", do you think you will get good responses and offer itself to robust analysis ?
Open ended questions are more difficult to code and analyse when it's 'textual'. Perhaps more details on what you are trying to study in particular or discussing on the questionnaire design might be more helpful for you. You can put up certain areas which you wish to analyse so others may offer ideas on how best to collect that data perhaps.
With all the trouble with open ended questions, and the difficulty in analysis, we will still end up with a result that may not be easily decipherable to the population we are studying.
If the open ended questionnaire has questions that can be categorised or quantified easily, then it will be easy to code and analyse, else I am not sure how useful a result we'd derive even with appropriate statistical analysis. I always search for pre-validated questionnaire which I'd translate / change according to the culture & language of the people. Then a small pilot study of a segment of population with a full analysis will reveal the issues at hand as well as increase validity of the tool. Once that step is done, one may proceed with the entire study.
Regards,
Dr. Akilesh. R
India
drakileshr [at] akilmaxfac [dot] com
Please see link.
http://www.hadassah-med.com/media/2016259/Thedevelopmentandvalidationofquestionnaires.pdf
I've just given a similar answer to another RG member.
As follows:
Validation is a long term process. It is concerned with the degree to which a questionnaire reflects reality. It is also about the questionnaire actually measuring what it is designed to measure.
It is always good to use validated questions from literature - directly with respondents or try to adapt them. Validity is linked to good questionnaire design, (such as inputs from experienced researchers, piloting with experienced researchers as well as respondents). The sampling frame and methods need to be robust, with appropriate rationale and determinations of sample size and, if necessary power.
Reliability is also an important consideration the degree to which a questionnaire will produce the same result if administered again, or the test-retest concept. It is also a measure of the degree to which a questionnaire can reflect a true change. As indicated already, there are a number of different aspects to validity. The degree to which questions within an instrument agree with each other is internal validity i.e., that a subject will respond to similar questions in a similar way. It also affects the likelihood of producing false positives and false negatives. The ability to make generalizations about a population beyond that of the sample tested is external validity. The degree to which the instrument can identify a true positive is sensitivity, for example, accurately identify a person who does have a belief. Similar to sensitivity is specificity, this is the degree to which the instrument can identify a true negative, for example correctly identifying the people who do not have the belief. Sensitivity and specificity are the flip side from internal validity. Related to internal validity is statistical validity, and assesses whether the differences in the questionnaire results between groups can appropriately be subjected to statistical tests of significance. There is also Longitudinal validity: whether a questionnaire returns the same results in a given population over time, assuming all else remains equal and in a social consideration there is Linguistic validity: whether the wording of the questionnaire is understood in the same way by everyone who completes it. The latter has implications when translating questionnaires for different languages and cultures.
Importantly there is content, construct and criterion related validity to consider.
Content measures the degree to which the test items represent the domain of the trait or property being measured. In order to establish the content validity of a measuring instrument, the researcher must identify the overall content to be represented. Items must then be randomly chosen from this content that will accurately represent the information in all areas. By using this method the researcher should obtain a group of items which is representative of the content of the trait or property that is to be measured
Construct must be investigated whenever no criterion of content is accepted as entirely adequate to define the quality to be measured, construct here explains some aspect of human behavior, such as physical ability, intelligence, or introversion.
Criterion is concerned with detecting the presence or absence of one or more criteria considered to represent traits or constructs of interest. One of the easiest ways to test for criterion-related validity is to administer the instrument to a group that is known to exhibit the trait to be measured.If the researcher has developed quality items for the instrument, a culling process should leave only those items that will consistently measure the trait or construct being studied.
Validity is also linked to Reliability (of which there a a range of types). There are Retest, Alternative Forms, Split_Halves and Internal Consistency methods. The latter finds wide use in Education and Social Science questionnaires
Excellent, comprehensive answer , Robert. Sam Messick gives the flesh to those bones.
- Badges
- Science method
- Similar topics
- Mathematical Sciences
- Statistics
- Applied Statistics
- Survey Design