Dear Raluca,

When we start a research we never manipulate our results.Rather we generally take this into consideration with the help of an alternate and null hypothesis that this assumption of a model being fit could or could not be true. so both the results are right.Like for example while the world famous VALS (Value and Lifestyle )model works well in the U.S.,Japan did a factor analysis on the model and foudn that it was not working for the Japanese customesr.So they devised a Japan VALS model specially for the Japanese.The idea of validating a model is to check if it works well with the target or not.So even if it is not working well,that itself is a research output.

Well it suggests that the scale does not perform the same way and /or the interpretation of it amongst musicians is different. You need to look at the items and the relationships within the scale. Does the data make sense? Can you name the 4 factors it is odd to have 9 factor solution as the whole point is to reduce complexity. Are there people who translate?

Raluca Matei

What do you mean when you write that the model was "unfit"? Talking about AMOS I assume that you refer to the indices which give you the fit for the whole model. These basically include all correlations between the individual constructs. Therefore it would make sense to have a look at each sub-dimension individually. The problem in your case seems to be that the individual sub-dimensions seem to be highly correlated (i.e., the individual items load on on other sub-dimensions). This is quite natural since you are measuring different sub-dimensions but at the end of the day you would like to have a clear data structure. One option would be to "purify" the scales by removing items, but this should be done with a different sample (or you split your own sample). Most likely, this will leave you with a completely different measurement instrument. I have co-authored two papers which you might find useful.

Article Statistical and judgmental criteria for scale purification

Article Exploratory factor analysis revisited: How robust methods su...

Hi Raluca,

This is the sequence for your 9 factrors.In case all your nine factors are constructs which I see they are you should have first done a reliability test on each of the nine scales.In case the cronbach for each scale was above 0.7 only then could you go for a CFA.If the reliability of the scale itself is not good,you need to change the items or rephrase the statements of the scale.Moreso you need to do a EFA before the CFA.In case you wanted to reduce these 9 factors to the crucial 4 or 5 then you should have done a EFA first and then done a CFA or SEM for model testing on the left over 4 factors.Even then a SEM can only be performed if the chronbach of the statements is 0.7 or above.

Another suggestion in case you took all these 9 factors and their relation from the past review ,there are very few chances that your model will not work.

Thank you so much for taking the time to help me with this!

Horst Treiblmaier, I did a one-factor CFA model which was fitted to the data for each of the 9 scales. I had poor scores for all relevant indices, such as Chi Square, degrees of freedom, GFI, TLI and RMSEA. As such, I went on to conduct an EFA and got stuck. I shall have a look at the articles you suggested - thank you!

Jaspreet Kaur, I did a reliability test for each of the 9 scales before doing anything else. All Cronbach's alpha scores were between acceptable and good. That is why I continued with a CFA. I noticed that relevant scores were poor, so decided to do an EFA. I don't necessarily want to reduce the scale to 4-5 factors; rather, that was what my scree plot suggested in the EFA. I checked this then with another CFA and it didn't fit at all. Not sure where to take it from here....

Katja Rudell, yes, there may be people who translate. My sample is formed of musicians based in the UK, but of various nationalities. How do you suggest that I look at items and their relationships within each scale? Also, why would I want to have fewer than 9 factors? Would it not be ideal to have exactly 9 factors, as this is how the scale is supposed to work (at least on non-musicians)?

It is fine it the model is not fitting.It just shows that the model is not valid for your target population which is also a research outcome.

Jaspreet Kaur, I have two questions in response to your last message:

1. Surely I should have done something simpler than to collect almost 500 responses just to conclude that the model is not fitting?

2. But then, what is the purpose of conducting EFA and CFA in this particular context, if my final conclusion were that the HLQ is not good for musicians? Should I not be able to adapt it to my own purposes via statistical manipulation and re-fitting?

MANY thanks!

Dear Raluca,

When we start a research we never manipulate our results.Rather we generally take this into consideration with the help of an alternate and null hypothesis that this assumption of a model being fit could or could not be true. so both the results are right.Like for example while the world famous VALS (Value and Lifestyle )model works well in the U.S.,Japan did a factor analysis on the model and foudn that it was not working for the Japanese customesr.So they devised a Japan VALS model specially for the Japanese.The idea of validating a model is to check if it works well with the target or not.So even if it is not working well,that itself is a research output.

Raluca Matei You raise a good point about the interpretation of the results. Strictly speaking, you should not play around with the data and report that your model did not work. Unfortunately, many journals do not accept "failures", which creates a huge bias in the academic literature. The best solution would be to split your sample and use one part for the creation of a model and the other part to test it.

If you do not have too many " problematic“ items, you can remove them (i.e., scale purification). Interestingly, this practice is very common in the academic literature in spite of the fact that you are adapting your data to the model.

Raluca Matei an exploratory factor analysis may give you the most parsimonious fit model versus a forced technique. Did you prespecify 9 factors? My main thing which I tried to bring across they try to reveal a latent structure that needs to make sense. So if you have 9 theoretical factors in health literacy and they make sense then the AMOS results are not helpful but maybe 4 factors are underlying the 9? Just a thought.

@Katja Rudell and @Horst Treiblmaier, thanks a lot! So, I have removed 3-4 items (out of 44) which seemed problematic in that they were reducing the Cronbach's alpha scores of their related scales. Then, I did an EFA which suggested I might have 7 factors. I looked at the Cronbach's alpha of each of the new 7 factors and those were at least .7, so all good. However, if I look at the items that loaded onto each of these 7 factors, not all of them make sense! So, although they seem to make sense statistically, face validity is rather odd...

Also, sometimes I have items loading onto 2 different factors but if the loading is .6 on a factor and .4 on another, I consider it as part of the former (the one with a higher loading).

I will now split my sample into 2 random samples and see if I find the same thing in both...

Might you suggest anything else?

MANY thanks!

@raluca matei just one more thing. Your musician sample is that post graduate degree I. E. Education may influence your health literacy. Ie if you had a higher educated sample this could act as an overall confounder and maybe also why the results don't match a general population sample. But I think you may be able to report out on the factors that come out? Kind regards and Happy psychometric day!! Well done for documenting. 😀

Katja Rudell, woaaaa, I had not thought of that! Good point! I shall investigate that further and look into how a more highly educated sample could affect factors...

Please see my previous response - I did find 7 factors after having removed some problematic items, but they don't entirely make sense...so, face validity is rather odd.

I should probably report the factors that come out and NOT also look at group differences (i.e. differences in health literacy by sex, professional level - student vs professional, instrument, education level, nationality, etc.), as these would be inappropriate given that they would be based on the original 9 factors... is this correct? Unless, of course, I create an entirely new questionnaire on the basis of the newly found factors.

Many, MANY thanks!

Raluca Matei I think you have received some really helpful advice up above from people who are obviously knowledgeable and willing to help.

Of course, factor analysis isn't an exact science, so you need to see what's the best way for the data to talk to you.

May I go back to some basics with you? I wonder whether you might use the scree test rather than the Kaiser criterion (eigenvalues greater than 1) to determine the number of factors to look for.

Also, I wonder whether you might try an oblique rotation. Also, suppressing items with loadings less than .40 might be a bit conservative at an early stage. I wonder whether there might be a little bit of insight residing in an item that, at an early stage, had a lower loading. Maybe try for a .30 cutoff early in the piece?

I could be wrong, but I would be reluctant to split your sample in half. There are various notions about what sample size is needed for factor analysis, but I like to work with the notion that the sample should be10 times the number of items. According to that, if you have 44 items, you'd need 440 people in your sample. There are times when fewer people might be OK, but that might be only when the factors are pretty "clean" to begin with - which yours don't seem to be.

And, sure, I wouldn't begin looking at group differences yet. Sort out your factors first!

I hope you enjoy what you're doing, even if, at times, it's frustrating.

Robert Trevethan you raised a couple of excellent points. I fully agree with you that factor analysis is not an exact science, given that so many different ways exist how the data can be analyzed. Oblique rotation is definitely a good idea. It might make the interpretation a bit harder, but after all we expect the various dimensions to be correlated, right?

Finally, I recommended to split the sample before I knew the number of variables. The sample:item ratio of 10:1 is a good recommendation and should be seen as the lower limit. I have seen factor analyses being published (in prestigious journals) with a much lower ratio, which makes the results questionable.

Horst Treiblmaier thanks for the support and reflection. I should apologise for not recommending your posts above, but I confess I didn't understand all of what you were recommending, so my innate "shyness" kicked in and I held back.

I, too, have often seen what I regard as inadequate samples - along with a range of other methodological practices that make me cringe - in prestigious journals. Unfortunately, it seems that reviewers (and editors) are often not up to scratch.

Horst Treiblmaier and Robert Trevethan , thank you immensely for taking the time to share your thoughts with me!

So, I did the following things:

1. Ran an EFA on the original, unmodified nine-factor scale and on the basis of the scree plot and the parallel analysis I conducted, with oblique rotation (given that some of my items seem to correlate above .3), there seemed to be between 4 and 6 factors (although eigenvalues suggested 7).

2. Next, I ran EFA with fixed (imposed) extractions of 4, 5, and 6 factors respectively.

3. Next, I checked each of the models above (with 4, 5, and 6 factors respectively) via a CFA. Scores were mostly poor.

4. Entirely separately, I re-ran a CFA on the original 9 factors (on each scale individually). Goodness of fit scores are good ONLY if I apply at least one modification (as suggested in the Modification Indices Tab in AMOS (Covariances) per scale. I know that this is not ideal (if at all acceptable!) and that I am essentially forcing my data to fit in with the original nine-factor model. However, I did this out of curiosity. In addition, I don't know what else to do at the moment...

Raluca Matei thank you for letting us know how you went. I was curious.

I confess that I'm not sure whether running a CFA on the same data as you have used for an EFA is a good idea or not. However, I have a hunch that you should get results that are somehow "parallel" if you did. That doesn't seem to be happening, so I can see why you are perplexed.

Have you gone back to inspect your data carefully? Is there narrow dispersion among the responses on many of the items? Are there outliers that should be attended to?

And when the EFA produced the 4, 5, and 6 factors, were they meaningful? Might they have become clearer and more meaningful if you (needed to) remove items with low loadings or with cross loadings?

I confess that I'm curious not only for your sake but also because a colleague and I are currently collecting data and I have no idea what factor structure might emerge or how clean that structure will be.

Of course, feel free to let us / me know how you go!

Raluca Matei

You are absolutely right as far as the modification indices from AMOS are concerned: if you use them, you adapt your data to the model. I know that this is frequently being done in research and more often than not it is not reported. Again, the removal of items does basically the same, but seems to be commonly accepted in the academic community (especially if only a small number of items are removed).

I am always a bit skeptical when it comes to highly complex scales with many subdimensions and items. It would be interesting to see what happens when you scramble the order of the items for the individual questionnaires...

Anyway, this does not help you much at the moment. I guess the best solutions would be to report all that you did and to present various analyses in you paper. Additionally, you can speculate whether it was the composition of your sample (see the comment from Katja Rudell above) which impacted the results. Of course, this would mean that your research goes in a different direction, but I would find that to be very honest and interesting.

Robert Trevethan and Horst Treiblmaier , I am so grateful for your support - you might not believe me, but your choosing to make time to help me means so much to me! I am in the process of running a few models as indicated by Eigenvalues, scree plots and parallel analysis (will try a few with 4, 5, and 6 factors respectively), as well as a modified (through covaried errors) original 9-factor model and see whether CFA indices are any good (am talking about an overall CFA for each of the models, rather than for each factor separately, as I've done so far. After all, I found a paper in which authors run a CFA for the same Health Literacy Questionnaire on their particular sample and while the original 9-factor model seems to fit with their sample, they provided goodness of fit indices for each of the factors separately and those weren't always good. Unlike me, they did not seem to need to covary errors, fortunately). I will be removing all items that cross-load, as you suggested (and rightly so!) and will be using oblique rotations throughout.

Absolutely, Robert Trevethan , I shall keep you updated! That's the least I can do in return for your kind support!

Thanks, Raluca Matei . I'm keen to see you succeed. Horst T., I'm sure, would be, too. I suspect he's a pretty busy person, but he might at least appreciate having an opportunity to glance over how you progress.

To battle!

Raluca Matei Good luck with your research!

btw: Covariating errors is something that I would not recommend. It means that you have systematic error variance and this is really hard to justify.

Horst Treiblmaier , agreed! I have dropped that option... the result was even worse anyway!

Hi Raluca,

I would suggest the following recommendations:

1) If your sample size is around 1000 or more, I would randomly split the sample in two subsamples: i) 40% training sample, and ii) 60% test sample.

2) The training sample will be used to conduct an Exploratory Factor Analysis (EFA), and the test sample to conduct a Confirmatory Factor Analysis (CFA).

3) Both, EFA and CFA need to be conducted using the correct estimation method. If your items are ordinal with few categories (i.e., less than six categories) you need to use the Diagonally Weighted Least Squares (DWLS) estimation approach.

4) For the EFA analysis, you should compare the model fit indices (e.g., CFI, TLI, RMSEA) for different models. You can consider start this process with a model with one factor and finishing with a model with nine factors. Then, you can choose one or two of these models with the best model fit indices. The factors for the selected models need to show a clear interpretation. In my opinion these analyses need to be conducted using correlated factors rather than orthogonal factors (i.e., varimax rotation). During this step, you can consider delete some items that show low loadings (e.g., lower than 0.30) or high cross-loadings (e.g., larger than .40 in more than one factor).

5) If you have a test sample, you can apply a CFA based on the models obtained in 4) using this new dataset. If you don't have a test sample you only can report the EFA final results. It is a bad practice to apply EFA and CFA to the same data.

Best,

--

Roberto

Horst Treiblmaier Roberto Melipillan Robert Trevethan Katja Rudell Jaspreet Kaur

Thank you again to all of you who responded to this! I have now contacted one of the authors of the HLQ who is a research methodologist who specialises in the application of survey methods to public health. I also obtained his permission to share his answer with you, as I thought this may be of interest. So, here goes:

"I take from your query that you were using the default confirmatory factor analysis (CFA) approach in Amos to fit your model. This uses maximum likelihood (ML) estimation which is *not appropriate* for the HLQ items. The HLQ items have 4 and 5 point ordinal response options. CFA (and exploratory factor analysis; EFA) of ordinal data with 4 and 5 point response options should be done with weighted least squares estimation and polychoric correlations* (the appropriate method for CFA is labelled WLSMV in Mplus and DWLS (diagonally-weighted least squares) in LISREL. Either one of these specialist structural equation modeling programs should be used to analyze the kind of ordinal response data generated by the HLQ.

That said, we have recently been using Bayesian structural equation modeling (BSEM) in Mplus to analyze HLQ data. I understand that from Version 7 onwards Amos has a Bayesian CFA option and this may be appropriate for use with the HLQ. The second edition of Barbara Byrne's book "Structural Equation Modeling with AMOS: Basic Concepts, Applications and Programming" (2nd Ed., 2010) has, I believe, information on the use of Bayesian SEM in Amos. Our use of BSEM involves setting informative (small variance around zero) priors for residual correlations and cross-loadings to provide some so-called 'wriggle room' around zero for these estimates. I doubt this option would be available in AMOS, however using Bayesian analysis in AMOS should provide you with a much more appropriate approach to CFA than the Amos default ML approach.

Using SPSS for exploratory factor analysis with HLQ items is, similarly, not at all appropriate. While I would strongly recommend that you don't immediately fall back on EFA without a thorough exploration of the reasons for any poor model fit in a Bayesian CFA analysis of HLQ responses, if you do need to do an appropriate EFA there is (or was) a free-ware program available on Prof. Michael Browne's home page at the Ohio State University. The software is called 'Comprehensive Exploratory Factor Analysis' (CEFA) and is very user friendly. 

Finally, there are many appropriate factor analysis tools in R for analyzing ordinal HLQ-type data.

*Polychoric correlations are actually based on the assumption that there is a normally distributed latent variable underlying the ordinal response continuum. As I understand it polychoric correlations handle non-normally distributed ordinal data more accurately than do Pearson correlations. In our situation with non-normally distributed ordinal responses to self-report items factor analyses using polychorics and an appropriate estimator (e.g. diagonally-weighted least squares) is the best option we have available aside from a full Bayesian analysis with small-variance priors. I’m not certain whether polychoric correlations are necessary with Bayesian analysis. Mplus has the option of declaring the data CATEGORICAL for a Bayesian analysis; I’ve experiments a little with this, but have found the results very similar whether or not this option is used and have not used it routinely. I don’t know whether Amos provides a polychoric option for Bayesian analysis."

So, based on his response, we have conducted a one-model CFA with weighted least squares estimation and polychoric correlations and goodness of fit scores look much better now. We still need to run a CFA for each of the nine factors/dimensions separately, as suggested by the same research methodologist...

Raluca Matei This is an excellent and highly competent answer. Thank you for sharing it with us. He is absolutely right in pointing out that the core of the problem is the use of 4 or 5 point ordinal response options. Interestingly, this does not stop a vast majority of researchers from using them as if they were interval. I was interested in measurement issues years ago and had a look at continuous scales and Item Response Theory, which builds on different assumptions than does Classical Test Theory. I also had a brief look at Bayesian Statistics. It is good to hear that more rigorous measurement and analysis are coming up in some communities.

Glad to hear, that your goodness of fit scores look better after using the more rigorous approach!

Raluca Matei. Thank you very much for getting back with how you fared. I really appreciate your having done so, and I notice that Dr Treiblmaier has already responded. This is true collegiality.

I'm glad that the recommended procedures seem to have worked much better for you. I am concerned, however, that your "expert advisor" has indicated SPSS to be inappropriate for analysing the HLQ data.

That makes me wonder whether SPSS is inadequate for factor analysing data from other scales. At the moment, a colleague and I are conducting research about teacher sense of efficacy (TSE) and I am sure that almost all factor analyses in that corpus of research are conducted with SPSS. I am critical of many aspects of the TSE research, but I now wonder whether I should add use of SPSS for exploratory FA to the list of my criticisms - AND, more worryingly, whether my colleague and I should look to do something else.

Thanks again.

I am having additional thoughts about factor analysis of Likert-type data given the advice you received, Raluca. I wonder whether problems arise when the data are based on only 4 or 5 response options, but whether those problems pertain when there are, say, 7 response options.

Any ideas appreciated.