Hello Ruggero,

if your model fit is bad, than the idea that those 24 items measure one underlying factor is wrong and the structure is much more complex. Of course you can add everthing together but that would imply a rather index-like composite instead of a latent variable. Reliability does not help you much, as measures of internal consistency require that all respective items measure the same factor.

Hence, there are two options

1) Respecify the model until it fits with all the involved caveats (e.g., capitalisation on chance) and implications (i.e. that the results are preliminary and semi-exploratory)

2) Create the composite, let loose of factor-like thinking, let loose of estimating reliability and accept the ontological lack of clarity of any aggregate index (i.e., you do not know exactly what's in it)

Best,

Holger

Hello Holger Steinmetz,

Thank you for your reply!

Regarding the first option, I tried to respecify the model with a factor division based on an EFA. Indeed model fit index were good. But I shouldn't run the SEM based on the Exploratory made with the same dataset, so it's not really good. I also tried to respecify the model with a factor division based on previous research, and all the model fits are bad. I am thinking of opting for a path analysis, which I think is what you suggest as the second option (if I'm not mistaken).

However, if I use the single item method, I saw that all the fit measures are good. So I was thinking of using this method for the variable that is causing problems, whereas using the multiple item method for the others. But am not sure that this is fine.

Thank you again and have a nice day.

Best,

Ruggero

You can perform a plotting method to adjust the variance if you take into account the factor loads of the indicators.

Hi Cristian Ramos-Vera,

thank you for your reply!

Do you mean that I use the factor loadings of the single items as loadings to the latent variable?

Thank you and have a nice day.

Best,

Ruggero

I don't understand why you think you should not run the SEM based on an EFA. If there is no prior theoretical basis for assigning your items to underlying factors, then EFA is your best option..

Hello Ruggero,

I totally agree with David's posting. I find it a bit odd to explore the structure and--despite the exploration--to go on with a bad fitting model to avoid the caveats which I mentioned. Yes, of course, exploration can lead to nonsensical results due to

a) capitalization on chance: You find something which is unique to our sample and not replicable

b) fitting a wrong structure (although the EFA resulted in a specific structure, that structure may be wrong, e.g. when the data generating process is no factor model or a more complicated factor structure.

However, the best way to deal with that is to report everything transparently and to discuss both issues above in the limitations section of the paper. Having said that, problem "b" above can be tested by incorporating further external variables into the SEM that would reveal if the structure is viable.

Of course, the single item model fits--there are no assumptions and implications left that you could test.

All of that does not mean that you have to force the model to contain latent variables, though. I often use composites when I am not really interested in the structure and the focus is on something different. This most viable in behavioural constructs which are in most cases aggregates of specific behaviours. But the downside is that you mix different things together. Imagine the following example: You do an experimental intervention on health and your invention comprises a) sports 3x per week, b) quit smoking, c) eating vegetables/fruits, d) 3 sessions of meditation. One month later you measure a significant increase in the respondents' wellbeing. What do you learn from this?

Best,

Holger

Hello David,

thank you for your reply. I was told in different sessions that is bad practice to use SEM after the EFA. However, by looking for references on this I found a post from 2014 in which you and Holger were saying the same things as now, so I feel pretty confident in doing it now. I just need to find a proper reference in case someone raises this issue.

Thank you very much for your help!

All the Best,

Ruggero

Hello Holger,

I see, thank you! As I just wrote to David, I found an old post in which you also say the same. So I will go like this.

Given that I have longitudinal data, I will run a CFA on the structure that I found at time 1 with the time 2 data and see how well it fits. If it does I think I should be fine. If not I will use a mixed technique.

Thank you again for your help!

All the Best,

Ruggero

"However, by looking for references on this I found a post from 2014 in which you and Holger were saying the same things as now"

Haha, David L Morgan , this is either a sign of high reliability or stubborness.

To the serious stuff: The discussion rather center about "doing EFA then CFA", where CFA is used to test a factor model structure that was prior explored. This is indeed somewhat problematic for obvious reasons (although not completely nonsense as you at least test something--namely those restrictions that are not involved in an EFA. That's why even in these support-guaranteed situations, the CFA models will often not fit although the EFA suggested the structure. The reason is that EFA is blind with regard to other structures than factor structures. It is like a hammer.. it is able to do only one thing....

Your idea of "replicating" the structure, sorry, does make no sense. I would not do this. Conduct the longitudinal model appropriately and spend your time on this. Then discuss the caveats which I mentioned so that everyone can judge whether to trust your results and everyone knows that replication (or better "enriched replication" what I defended in the posted link below) is warranted

https://www.researchgate.net/post/Can_I_use_EFA_before_SEM_analysis

Best,

Holger

In your case, the EFA was only a preliminary step and the goal is the SEM and the relationships

Hi Holger,

I see. I will, thank you!

Thank you all for your help, appreciate it!

All the Best,

Ruggero

If you have theoretical bases for multiple dimensions of the same latent variable. it's better you should go for higher-order factor loading.

If you have a theoretical model means you can proceed with Confirmatory Factor analysis. One more thing 24 measured variables you should not load in a single unobserved factor. For that, you need a strong supportive point. Otherwise, you have to do factor analysis then EFA later SEM….

Holger Steinmetz I agree that EFA is like applying a hammer to the data, but sometimes a hammer is not just the only tool but the best one. What else is one to do if there is no prior theoretical basis for assigning items to factors?

And what if one looks back in the literature and finds that the well-ordered assignment of items to factors from prior research was due to EFA -- should one then ignore that?

Dear David L Morgan , sorry, this will be long :)

Short version: EFA will work IF the data generating process is a factor model and the assumptions of local dependence hold. Often, the process deviates from this ideal with increasing biases.

How I see the literature using EFA? At least as a candidate for rigid replication

Loooong version:

As said, I have nothing against EFA per se--only against the uncritical and almost reflexive application of factor-thinking in general without even considering alternatives.

EFA will work well if the data generating process is indeed a latent variable causing the indicators fulfilling the necessary causal assumptions (i.e., local independence) but will have problems the more deviations will exist from this ideal. This will be the case

a) when you have single indicators for which the algorithm is unable to identify a factor (as a factor is inferred by the correlations of the convergent indicators). The same is likely for very small sets of indicators that measure a latent variable. In this regard, Hayduk and Littvay (2012) argue that "exploratory factor analysis is not likely to locate half as many latents as indicators, or indicators clustered in tidy pairs" (p. 2).

b) The indicators influence each other (Bollen & Medrano, 1998) which is very likely in questionnaires. In this regard, I find it almost funny which naive process thinking especially psychologists have (i.e., perceiving a respondent as having some kind of amnesia) and given the huge literature in survey research about sequence effects and the way respondents interpret the meaning of questions based on prior questions (Sudman, Bradburn & Schwarz, 1996). I learned especially from sociologists the importance of cognitive pretesting interviews in which you (I guess you have some experience with that) rub your eyes how individuals interpret questions.

c) Whereas case "b" at least has some kind of underlying factor, the correlations among the indicators may be due to some kind of totally different process and there is no factor at all involved. In this regard, there is an increasing literature in the last decades that use network models as an alternative to factor models (Robinaugh et al., 2019).

One important point in all of this is to grasp that "factors" are not the same thing as "latent variable". Equating both is again a result of the "factor-reflex". For instance, the mentioned net work models would (IMHO) surely involve latent variables (that is, each item is a error-containing reflection of some underlying true entity) but there is no common factor (defined as a common cause of at least two indicators involved.

You asked how I would evaluate the masses of literature that rests on EFA? At least as a candidate for solid re-assessment and replication studies.

To be more bold (as stated hundred times on RG), I see no special value in doing EFA despite in situations in which one absolutely has no clue about the items. In these situations, as any exploration and data-dredging, EFA at least may give you some ideas. The problem however is rather that the researcher stops at that step. What I see critical is the cargo-cult ritual consisting in

a) create many items

b) run an EFA

c) trow away items with bad loadings

d) compute alpha

e) add items together and you have a scale.

f) never challenge that

Agreed, step b is very often substituted by conducting a CFA which involves the steps

b1) conduct a CFA

b2) test against a one-factor model (which never fits)

b3) explain away the is fit by parroting the phrases "models never fit", "sample size leads to misfit" and the "model is ok/acceptable")

The reason for all the problems is the undying conviction that one has to have maaaany indicators. It is astonishing that you will not find such a conviction in other sciences. I already hear the counterarguments "but measurement error" and "but behavioral constructs are more abstract". Fair enough. But: every measurement procedure involves error (see the discussion about the corona tests) and there are a) other procedures to account for errors, for instance with fixed error variances, or instrumental variables (the econometricians' approach). And even only two indicators (I would always try to identify 2-3 crystal clear ones) allow estimating an error in a latent variable framework, and sometimes, even one indicator embedded in a SEM allows that.

With regard to the "constructs are abstract" argument: Well sometimes they are abstract and sometimes they are very precise. But the question is what "abstract" means. I understand it as "generalizable across situations/targets" vs. "situation/target specific". On what level you conceptualize your construct depends on the research model (see the work by Icek Aizen about the correspondence principle which nails that).

But even abstract constructs do not imply the necessity to have my indicators.

For instance the very global personality trait conscientiousness may be measurable by one single indicator. This could be "overall, I am a very conscientious person". Whether this is a good measure depends on the very formulation and quality of the indicator and I would rather see people spending their preparation time creating these 1,2 or perhaps 3 crystal clear indicators. You see the same logic and indeed its applications in measures of job satisfaction that can be measured by a single indicator even in the most abstract case ("overall, I am very satisfied with my job", Wanous, et al., 1997) or regarding specific aspects ("I am very satisfied with my pay").

Now I hear again the next counter argument "but concientiousness is much broader and contains several things" (e.g., being dependable, organized etc.). And now we get to what should be "conceptualization work" (instead "measure construction work"--namely to think hard about whether conscientousness is an umbrella term of these facets or again a common factor ( Ruggero De Agostini , now you can start reading again :). If it is an umbrella term (or formative / aggregate construct) than, yes, you need items for all facets. But in this case we again approach the problems of the factor-reflex as such sets of measures are nonetheless being forced in a common factor model. As the alternative, the factor model may indeed be correct and conscientiousness is indeed the common underlying cause of dependability etc. In this case it is simply not necessary to include all items. The reason is pure logic: If a set of k items follows a common factor model and all k items are valid measures if the factor than, consequently, all possible permutations of smaller sets of items (up to single items) is also a valid measure of the factor. With one item, you have a potential error estimation problem but that does not make it invalid.

However, people still have the belief that the latent factor is some kind of result from the indicators and that's why many of them are necessary. But a factor as any other latent variable is just one singular dimension, it has no breadth or facets. It is just a line of numbers that reflect some quantitatively existing, and appropriate entity "out there".

Psychometrics has made a business out of describing measurement as something complicated and printed tons of books describing the aforementioned ritual . Actually, it is really simple: Once you are clear about the entity you want to measure, create THE very measure that would reflect it if it is the thing you belief it is. This philosophy leads to realistic structural model that still has masses of testable implications, but a chance for success for a fitting model is given. In cases it does not fit, you have a chance to find the needle in a haystack and to come up with a reasonable respecification. If the model fits you have

a) a clear, precise construct oriented in entity realism (vs. abstract blurry "constructs")

b) the selection of likewise crystal clear reflections of that construct (because you have some idea about the causal measurement process, see the criticism on the lack of thereof by Borsboom, Mellenberg and van Heerden, 2004).

c) a possible fitting model with

d) strong factor loadings (again implying validity evidence and increasing the chances to falsify the model if it is wrong).

Models based on the psychometric ritual

a) are conceptually often very blurry (rather based on a constructivist philosophy)

b) are based on vague measurement processes leading to a shotgun mentality of including many items ("something will hit")

c) Misfitting models with no chance to respecify them (beyond including masses of error covariances)

d) weak loadings making it unclear what you actually measured.

Perhaps this rant went far beyond what you asked but EFA is just an approach that has it place in a paradigm. It is not that EFA is the problem, it's the paradigm.

Best,

Holger

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.

Robinaugh, D. J., Hoekstra, R. H., Toner, E. R., & Borsboom, D. (2019). The network approach to psychopathology: a review of the literature 2008–2018 and an agenda for future research. Psychological Medicine, 1-14.

Hayduk, L. A., & Littvay, L. (2012). Should researchers use single indicators, best indicators, or multiple indicators. BMC Medical Research Methodology, 12(159), 1-17. doi:10.1186/1471-2288-12-159

Bollen, K. A., & Medrano, J. D. (1998). Who are the spaniards. Social Forces, 77(2), 587-621.

Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers. San Fransciso: Jossey-Bass Publishers.

Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82(2), 247-252.

And CFA will solve all that? I sincerely doubt it.

Dear David, my point was not so much that EFA is bad but that most social scientists do not consider alternative structures as potentials. EFA can identify factors and sets of convergent indicators, if all assumptions hold and I argued that this is often unplausible. CFA (or better SEM) is more flexible and allows you to specify the structure as you like. The new network models add nice possibilties to the tool box. So we do not have only a hammer but also a saw and a scewdriver :)

All the best,

Holger

There is no problem if you combine both single-indicator and multiple-indicator factors in a model. The most important thing is to be able to justify your choices. I am afraid a factor with 24 indicators is obviously problematic. I will advise you to check the unidimensionality assumption of the scale using either EFA or CFA. The latter is preferred. Through this analysis, you should be able to detect indicator(s) that worth being removed from the scale, provided it makes sense conceptually. Then, you can now proceed to evaluate your structural model. This two-step method of evaluating the measurement before evaluating the structural model has proved useful in dealing with your likely problem.

The fact that CFA is more flexible just means that you can "play around" with assigning links between indicators and underlying factors. I don't see how that is an advantage over having an underlying mathematical basis for at least an initial assignment.

I certainly agree that under various extreme circumstances, EFA will experience problems. But I generally start by advising that someone try methods that work for most people most of the time. If the results are nonsensical, then we step back and look at the underlying data to see if there serious problems.

And none of this answers the original question: If one began with large number of items that failed for form a single scale, how should one proceed from there? Nothing about CFA gives you any indication about how many factors might be necessary, or how you should reassign items as you move from lower numbers of factors to higher numbers.

Of course, the best answer is to not get into this kind of situation in the first place, by beginning with a hypothesized measurement model. But when that isn't an option, then at least a preliminary EFA is the best available option.

Sure, I totally with you David L Morgan on the fact that a best practice when the CFA fails completely is to conduct a preliminary EFA. However, I may disagree with you on the notion of "play around" with a poorly fitted CFA. Every post hoc analysis in CFA is guided not only with a statistical argument but also with conceptual appropriateness. After all, the CFA I know is meant to test a hypothesised model that is based on established theory.

The original question is about the lack of fit when a factor with 24 indicators was included in the structural model. It is very rare for this to happen had been the analyst Ruggero De Agostini evaluate a measurement model, of course with a CFA, for this factor before he proceeded to the structural model. The CFA will detect any local or global misspecifications at the initial stage. Then, it is left for the analyst to decide on what to do with the scale either modify it (if it makes sense conceptually) or conduct an EFA. The only fear with the EFA is that another independent sample is required for further analysis (hypothesis testing using SEM) in line with the best practice.

I think the "conceptual appropriateness" of a self-specified CFA is initially in the mind of the beholder. Then the goal is to convince an audience that your assignment of items to factors makes sense from their point of view as well as your own. But that really isn't any different from doing an EFA and claiming that results make "intuitive" sense.

As for local or global mis-specification, that typically only provides limited information on how to improve the model (such as adding a cross-loading or a correlated error). It is very unlikely to help you with the case here, which began with a poor fit in assigning 24 items to one factor.

And you are right that it would be best if one had a completely different sample to replicate an EFA, but second best is to randomly divide the original sample and run separate EFAs.

Dear David

"playing around" is a bit of an extreme formulation. But even if one does that, what's wrong if the result is reported as an outcome of an exploratory CFA approach? Besides that, playing around in a CFA context involves at least some intention or thinking which EFA does not.

When you expect deviations from a the factor model as "extreme circumstances", I expressed some doubt that this is true and I presented theoretical considerations and empirical results that also question that. In my experience, 95% of CFA models don't fit--that's a further support that often, factor models could be at least undercomplex. But again, an EFA might be a starting point but it is important to enlarge the model by new validation criteria (leading to new testable implications) and, thus, challenge it. That is almost never the case (see the "psychometric ritual".

And yes, the "conceptual appropriateness" is in the mind of the beholder. This is the core of model generation: The scholar expresses his/her ideas about the meaning of the variables and his/her causal assumptions. These ideas can come from anywhere (formal theory, intuition, or exploratory approaches). The point is to present these ideas crystal clear and to test these ideas. That's why I strongly object rules of thumb, e.g., regarding factor loadings: The loadings must by evaluated with regard to their theoretical appropriateness. And again: Others may disagree with his/her interpretation, challenge them, or provide a different model. Business as usual.

I further disagree with your pessimistic evaluation that a misfitting model (such as that by Ruggero De Agostini ) is lost and cannot be improved. Every SEM software prints estimates of local misfit--the specific correlations among indicators which the model failed to reproduce. This information combined with a theoretical re-assessment of the indicators can often result in a eureka insight and improvement of the model. If Ruggero would provide the question wordings and the covariance matrix, we could probably could come up with an improvement. I am sure that a simple look onto the question wording would lead to a model that is at least dramatically better than the simplistic one-factor model. My preferred way, however, would be to start with the very entities of interest and their definition and than to select the key indicators. A doctor would not start by using all available measures but with the thing s/he wants to measure. "Blood pressure ? Lets take a measure for blood pressure".

And finally, I really doubt the claims about the need of a new sample to evaluate the EFA model. I wrote dozens of postings with arguments and simulation results to show that a simple replication of a model (no matter which structure it contains) will surely help regarding some issues (data errors, capitalization on chance) but not with regard to validating or supporting the inherent causal claims. If you think that X causes Y and you find a correlation, a replication that again finds that correlation does not support the X-->Y effect. The key is enlargement and challenge.

Best,

Holger

What is wrong with shall we say "experimenting" to find a model in CFA is the danger of over-fitting the unique features of a given data set.

And again, the CFA/SEM reporting of "local" problem with misfit will be of little use in moving from a single factor to a multi-factor model. If one didn't have a theoretical model to start with, then hoping to find one through local misfit corrections isn't very likely.

Finally, are you saying the 95% of the time you have found that CFA results don't match initial EFA findings? I certainly don't encounter that when I do EFA with Maximum Likelihood extraction and oblique rotation.

Dear David,

yes, experimentation (=exploration) can lead to many problems (i agreed to that several times) in that tread. So no objection here.

We're going in circles: The report of misfit alone tells you nothing, they are the "traces at the crime scene" which the detective has to interpret based on this theoretical background. It is simply abduction or inference to the best explanation.

There was a misunderstanding regarding the 95% that did not involve prior EFA. It is just the number of CFAs that I saw and conducted by myself.

All the best,

Holger

Holger Steinmetz Check on youtube James Gaskin - a really good resource for SEM. Also he has a wiki tool gaskination. I found him tutorial very helpful, along with the tips/addons as well. Hope this helps

That is good advice.

Dear George Onofrei ,

I know some of Gaskin's videos. I am full of respect to anyone you spends time and effort to contribute something to others but I have also to admit that I would not concur with some of the things he tells and recommends. I could imagine recommending the videos to newbies to learn the technical way to do SEM in AMOS (beyond trying to convince them to avoid AMOS generally and to use R :) - but not with regard to substantial things he says or recommends. As I said, no deep criticisms involved just a slightly different view on things (as presented in this thread).

This also true for the Book by Barbara Byrne which had contributed substantially to so much malpractice regarding SEM, especially since it is the bible for AMOS users.

Beyond that, I did not really understand the reason for mentioning Gaskin's video with regard to the topic discussed in this thread....

Best,

Holger

Sorry H, the answer was for a different post; i was reading your comments and when i went to reply, by mistake replied to yours. Indeed, i agree with your point about the videos, however James's work is good for someone who is just starting SEM. Thanks