so I came across a situation where a cross sectional Servery based research was done with a questionnaire that they designed themselves
and they finished the data collocation and have collocated the whole target sample
The Research team didn't do a pilot study.
When they wanted to start the analysis they wanted to do cronbach alpha to measure reliability.
- Cronbach alpha happened to be 94% (showing excellent reliability).
- but they have only done face validity for the questionnaire, and didnt do anything else like Principal Component analysis [PCA].
Q1/ so can they just go with the flow and write in the methodology, and result section that they have done cronbach alpha and it showed great result of etc etc... ??
Q2/and can they say that the questionnaire is a valid questionnaire ??
Omar,
It seems that there may be some degree of confounding of validity and reliability here. There are many issues regarding differing forms of reliability and evidence for validity one might consider, but I will try to address the specific questions posited here.
First, if there is adequate evidence for reliability in the "whole" sample, other than the added comfort of replication, that this was not done on a pilot sample seems of little concern. Indeed, the reason we typically do a pilot testing of an instrument is to avoid getting to the "whole" sample and having inferior psychometric characteristics. If lucky enough to have what we need with the total sample without a pilot sample, fine. So I see no reason why the answer to your Q1 is not yes. But ...
I don't know how many Ss were included in the sample upon which coefficient alpha was calculated. Coefficient alpha is not dependent on N, but the precision with which we can estimate it is. Therein, I recommend considering the sampling variability of alpha. This has been considered in a few ways, but a method I recommend is the ASE (check Duhachek for references). Another issue that I am fond of considering is instrument efficiency. Although I take it that the coefficient alpha is .94, depending on the number of items in that rendition, questions about whether there may be subsets of items that afford as much, or more, reliability and ASE may be of interest.
You comment about face validity and examination of dimensionality regards validity not reliability, as does your Q2. Validity is an entirely different matter and can be investigated in a variety of ways, and does depend on the kind of validity of interest. Some broad-brush possibilities regard consideration of dimensionality, to which you allude, through exploratory methods (but I would not recommend PCA), or, if theory is ready for it, confirmatory methods. Then too, if possible predictive validity is often most convincing. This demands a tome unto itself:-)
JDM
It is correct you should only present evidence of validity according to your objectives that are based on the inferences of the scores obtained from your study. You should consider for the evidence of internal validity the confirmatory factor analysis.
I don't think that either Principal Components Analysis or Confirmatory Factor Analysis is a very effective technique for assessing validity. What they investigate is whether a given set of items are uni- or multi-dimensional, which is only a limited sense of whether a measure performs in the ways that it was intended.
Also, just from a practical point of view, it is hard to get an alpha as high as .94 and have a multi-dimensional measure. But if you do want to explore that possibility, I would suggest using Principal Factors Analysis instead of PCA and doing an oblique (correlated factors) rotation.
thanks John for all the details and the Advice.
John Morris
AND thank you too Cristian Ramos-Vera and David L Morgan .
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire