We are intending to measure the awareness, knowledge and attitude. So we have created a quantitative questionnaire to measure these items. The questionnaire consists of 3 sections (demographic information, questions about awareness and knowledge, and questions for attitude). All questions are multiple choice either yes/no/not sure or to choose the correct answer for knowledge awareness and to choose the correct/positive attitude for attitude questions.
For this matter I have few questions regarding Cronbach's alpha test:
As I read from the literature the Cronbach’s alpha is affected by dimensionality. What is the meaning of dimensionality? Can I consider my tool as multidimensional tool since it contains questions regarding demographical information, awareness and knowledge, and attitude? How can I measure the dimensionality? Can I use factor analysis and how?
For Cronbach’s alpha what is the required sample size? Should I choose all questions (demographical information, awareness and knowledge, and attitude) together or test the questions for each section separately?
I think you may need to differentiate between questionnaire and instrument.
Questionnaire can be used in survey and the items cannot be added up to reflect a single concept, like attitude or quality of life.
Instrument, however, required a stringent testing process to ensure the items and its response modality that are able to reflect a single concept or a single but related factor (e.g., subscale).
Therefore, demographic data (e.g., gender), most of the time, will not be an item of instrument. If the multiple choice questions are defined with a designed correct answer, it may not be suitable for a factor analysis and Cronbach’s alpha. This is because these items are not able to reflect the trend (like how satisfactory, how frequency, the degree of “agree to disagree”). So, my suggestion is 1. Go back to basic and review the need to develop an instrument for measuring the captioned; 2. Read more about the design of survey and what is instrument in social science; reconsider your need to validate this questionnaire; 4. Dimensionality is tested by RASCH analysis (i.e., item response theory) which I am still learning what it is.
I think you may need to differentiate between questionnaire and instrument.
Questionnaire can be used in survey and the items cannot be added up to reflect a single concept, like attitude or quality of life.
Instrument, however, required a stringent testing process to ensure the items and its response modality that are able to reflect a single concept or a single but related factor (e.g., subscale).
Therefore, demographic data (e.g., gender), most of the time, will not be an item of instrument. If the multiple choice questions are defined with a designed correct answer, it may not be suitable for a factor analysis and Cronbach’s alpha. This is because these items are not able to reflect the trend (like how satisfactory, how frequency, the degree of “agree to disagree”). So, my suggestion is 1. Go back to basic and review the need to develop an instrument for measuring the captioned; 2. Read more about the design of survey and what is instrument in social science; reconsider your need to validate this questionnaire; 4. Dimensionality is tested by RASCH analysis (i.e., item response theory) which I am still learning what it is.
Thank you Dr. Simon for your answer and advice.
we intend to use questionnaire in survey (not instrument) to measure a public awareness, knowledge, and attitude. The majority of questions were adopted from previous literature. We need to be sure that our questionnaire is measuring the concepts correctly by testing its validity and reliability.
Welcome. For such, you may simply check whether the items (developed by you) are relevant and adequate to measure the said concepts by a panel of experts. Besides, if you have time, try to ask your item to the target respondents for checking the comprehensibility, which means whether your items are clear enough to correctly comprehend by the respondents. Survey aims to here...
Hello,
I would definitely not test demographics for internal consistency (reliability). The idea of internal consistency is if all items included test exactly the same construct. Hence it is not a good idea to include gender, employment status, annual income and other background variables in reliability tests.
Also I would hardly be surprised if knowledge, attitude, and awareness 'thrown into the same sack' for analysis would yield high cronbach's alpha scores. They could be tested separately. If you are developing a new questionnaire then exploratory factor analysis would be applicable.
Regards, Witold
Cronbach's alpha is used to test a scale composed of multiple items. Do you intend to combine multiple items into a single score? If so, you need to think about why you are including each item in the composite score.
For instance, why would you include gender into a composite score?
Even though you are asking about attitudes, does it make sense to combine them into one attitude score or do you need more than one score. That is, are you measuring more than one attitude with your items? Factor analysis can help you to determine this. It may make sense to have sub-scale scores that are then combined into a comprehensive score. The same is true of the knowledge score; you may have sub-scales that you then want to combine into a comprehensive score.
There are a number of ways to test reliability including test-retest. Test-retest would be particularly appropriate for your demographic items since they should not change [age may change if they have a birthday between time 1 and time 2 so you would want to also have birthday]. You can also look at consistency between items such as gender and having had a pap smear or current smoking status and number of cigarettes smoked per day. Cronbach's alpha is designed to determine internal consistency of attitude scales.
For validity you need to first look at face validity. That is, does it look like the item measures what it is supposed to measure. For this you want a panel of experts to review the items. Then you need to look at whether the attitude and knowledge scores are consistent with those things that theory says that they should be consistent with and not with those that it says they should not be.
I agree with Wihold that you do not want to combine your demographic variables into a scale because they do not measure a single concept.
I also agree that knowledge, attitude, and awareness should be three separate measures. In particular, this is what is meant by "dimensionality" -- in the sense that these would be three different dimensions, so each should be its own scale.
As for a test-retest approach, this requires a lot of additional effort and would add little to your assessment of the reliability of the knowledge, attitude, and awareness scales, compared to just using alpha on your existing data.
Factor analysis is usually carried out when you are trying to create a instrument.There are two main aims of carrying out factor analysis.
1.when you want to categorize the items you have into clear definable domains.
2.To have the minimum number of items that represents each domain.
Also another thing to keep in mind is the target sample. If you test the reliability of the instrument with a certain sample of subjects remember that the survey then cannot be administered to these same subjects again.
Hence calculating the appropriate sample size according to your outcome is also important.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics