I am in the middle of questionnaire development and validation processes. I would like to get expert opinion on these processes whether the steps are adequately and correctly done.
1. Items generation
Items were generated through literature review, expert opinion and target population input. The items were listed exhaustively till saturation.
2. Contents validation
Initial items pool were then pre-tested with 10-20 target population to ensure comprehensibility. The items were then reworded based on feedback.
3. Construct validity
a) Bivariate correlation matrix to ensure no items correlation >0.8
b) Principal Axis Loading with Varimax Rotation. KMO statistic >0.5. Barttlets Test of Sphericity significant. Communalities less than 0.2 were then removed one-by-one in turn. Items with factor loading high cross-loading were removed one-by-one in turn. Then, item with factor loading 0.4). Next analysis is only on 4-factor model.
c) Next, i run Principal Component Analysis without rotation on each factor (there are 4 factors altogether), and each resulted in correlation matrix determinant >0.01, KMO >0.5, Bartlett significant, total variance >50%, and no factor loading 0.7, while overall realibility is 0.8.
e) I run bivariate correlation matrix and found no pairs correlation >0.5.
f) Finally, i am satisfied and decided to choose four-factor model with 17 variables and 4 factors (each factor has 5,4,4,4 items), and each factor had at least 3 items with loadings >0.5. Realibilility for each factor >0.7 while overall is 0.8.
.
My question is, am i doing it right and adequate?
Your response is highly appreciated.
Thanks.
Regards,
Fadhli
My advice if you can find a questionnaire close to the purpose of your study in the literature, it is better to use it rather than create a new questionnaire. Creating a new questionnaire is a demanding job, needs a team with experience in the construction of questionnaires and conduction of validity and reliability tests. It also needs piloting and significant revision.
Samy Azer : Thank you for you response. I did, but unfortunately I cannot find any related one.
Dear Fauzi,
The Pearson correlation gives the linear relationship between two continuous variables. Therefore, if you have any questions with continuous variables you can use Pearson correlation.
Where, the Spearman correlation evaluates the monotonic relationship between two continuous or ordinal variables. If questions (variables) tend to change together or constant rate can analyse with Spearman correlation test.
Principal Component Analysis (PCA) is a dimension-reduction tool that can be used to reduce a large set of variables to a small set that still remains the most of the information. It’s a multivariate technique which is used to determine the effect of group of variables (questions) together. It can be used either without rotation or rotatable variables. Even you can use cluster or factor analysis as PCA and derivate similar results according to the correlation between variables.
For items generation, you already went through the literature review, expert opinion and target population input. Did you go through the study area or any criteria to filter your respondents?
For the fully development of questionnaire pilot survey is needed to evaluate and to understand your population. Therefore, after finalizing the questionnaire initiate with a small group of population which represent all variables. Then you can realize what would be changed.
How did you get number of PCs derived from PCA? hope you used eigenvalues. And why did you use both rotatable and non-rotatable techniques. Think varimax rotation gives reliable results than non-rotatable analysis.
Dear Yohan Mahagamage ,
Thank you for your response.
1. For items generation, i am not sure what do you mean by 'go through study area' and 'criteria to filter respondents'. Can you elaborate further?
2. For construct validation, this is my steps:
(a) I started with Pearson correlation to filter any correlation >0.8 (as some authors advise to remove one of the pair which has high correlation which indicate multicollinearity), and there is none.
(b) Then, I run Principal Axis Factoring with varimax rotation, and based on the result:
- i remove items with communalities less than 0.2, one-by-one, and re-run (step repeated until no communalities less than 0.2)
- then i remove items with factor loading 75%)
- then i suppress items with factor loading 0.5 and total variance>50%, but four out of six factor only had only 2 items (some author suggest to have at least 3 items with factor loading >0.4)
- So, i try to run 0ne-factor, two-factor, three-factor, four-factor, five-factor and six-factor model, and found that only four-factor model satisfied the condition
(c) Then, I run Principal Component Analysis on the retained factors and found that four-factor model is most stable (based on eigenvalue, it yielded six-factor model , but four factor had only two items).
(d) Then, I test for reliability, and each factor had reliability >0.7.
I refer to this working paper and technical report, but I am not sure whether i am doing it right.
Working Paper Advice on Exploratory Factor Analysis
Technical Report Advice on Exploratory Factor Analysis
Hi,
For validation purposes the recommended statistical approaches are: 1) Exploratory Factor Analysis (EFA) followed by a Confirmatory Factor Analysis (CFA) or 2) Item Response Theory (IRT).
Principal Component Analysis (PCA) Is not the best approach for psychometrics purposes.
Roberto
Most of the time we adapt the instruments developed by previous study and modify to suit the study at hand. Sometimes we need to create and add few items to the existing modified items.
For validation procedure, there are few steps involved namely Pretest where expert comments are required for content validity, face validity and criterion validity. Next step is to gather pilit study data and perform EFA. Finally, collect field study data and perform CFA.
Dear Roberto Melipillan and Zainudin Awang ,
Thank you for your responses.
I am trying to conduct EFA, however I am not sure whether I am doing it right.
It is a bit confusing which method to use (Principal Component, or Principal Axis Factoring, or Maximum Likelihood, or others), when to use correlation or covariance for analysis, when to extract by eigenvalue or fixed number of factors, when to remove communalities / low factor loading / high crossloading / etc, and so on.
Do you have any suggested reading material that I can refer to particurly on the concept and step-by-step analysis?
Thanks.
Regards,
Fadhli
Mohd Fadhli Mohd Fauzi, there is quite a good post on RG by Peter Samuels titled Advice on Factor Analysis, dated 2016. There is also a useful article by Costello and Osborne, in 2005. Its title starts off with "Best Practices in Exploratory Factor Analysis". Please let me know if you need more details. Both of these references can probably be obtained by a quick Google search. Please feel free to get back if you can't, and I'll give you more specific details.
The two items above are not consistent in their recommendations. I am by no means an expert, but I recommend principal axis factoring (PAF) as the extraction method, using an oblique rotation (e.g., oblimin), and basing the number of factors extracted on the scree test, not the number of eigenvalues greater than 1.
Look at your pattern matrix. Once you have determined the number of factors that are likely to be in your data (from the scree test), run the FA again and specify the number of factors you want extracted. If the output is messy, perhaps try a different number of factors because the scree test isn't always clear.
Then, I recommend you initially retain only the items that have loadings above, say .35, and run the FA again. See whether you need to delete any items - and keep doing this kind of thing until you seem to have clean factors that each have items with loadings above, say, .40, and no cross loadings that are less than .20 in size.
I hope that helps. And please remember that there is a bit of an art to making decisions in factor analysis.
Feel free to get back with questions - or announcements of victory. :-)
Dear Robert Trevethan ,
Thank you for your response.
I already did step-by-step based on paper written by Peter Samuel as followed, but I am not quite confident as I just follow without depth understanding on the concept.
Working Paper Advice on Exploratory Factor Analysis
Technical Report Advice on Exploratory Factor Analysis
I will read on paper by Costello and Osborne and get back here with victory, hopefully ;)
Article Best Practices in Exploratory Factor Analysis: Four Recommen...
On other related matter, I would like to ask RG members on EFA for Likert-scale questionnaires. Some authors suggest Rasch, but most of the time I found that author use SPSS / AMOS. What is the most recommended method for EFA for Likert-scale questionnaires?
Regards,
Fadhli
Mohd Fadhli Mohd Fauzi, I'm glad you've had access to some authoritative advice, but please do notice that that advice is sometimes contradictory. It might be good to do your FAs several ways (it's pretty quick - and fun - in SPSS), and see whether you get the same results with different methods.
I suspect that what you're doing is OK. Others might like Rasch analysis, but I'm happy to do something else, more like what you seem to be doing.
SPSS is good for what you want so far. AMOS would be necessary for confirmatory factor analysis, but you probably don't want to do that.
One other thing: Are you sure that you have a sufficiently large sample? If you have too small a sample, your results could be dodgy.
Dear @Robert Trevethan,
I agree with your opinion that there is sometimes contradictory advices among authors, and that is why I am digging for more information to understand the concept and arguments.
My sample size is around 380. I am using 25-items 7-points Likert-scale questionnaires.
Regards,
Fadhli
Mohd Fadhli Mohd Fauzi, ah, good. I'm glad you have 380 participants for your 25 items. And I'm glad you're using 7-point options. I think that number is often best.
Thanks for setting my mind at rest for you.
As I suggested above, maybe try different methods, and then, when you report what you did, indicate the source of why you did things as you did.
Robt.
Hi,
Here you have some suggestions about how to apply EFA/CFA with your 7-point ordinal categories data:
1) You can analyze your data assuming they are continuous. In this case, the recommended estimator is Robust Maximum Likelihood (MLR). This approach corrects for non-normality in your data.
2) You can analyze your data taking into account their ordinal nature. In this case, the recommended approaches are one of the Diagonally Weighted Least Squares (DWLS) estimators: i) Mean and Variance Weighted Least Squares (WLSMV), and ii) Mean and Variance Unweighted Least Squares (ULSMV).
3) Regarding the number of factors, you can identify this using : i) CFI, ii) TLI, iii) RMSEA, and iv) 90% CI for RMSEA.
4) Finally, because your sample size is not too large (n = 380), you can only apply EFA. To conduct a CFA to confirm your EFA results you will need a new sample.
Best,
Roberto
- Badges
- Science topic