I don't see any conflict.

if your sample is large enough you could divide it into two causal subsamples and perform the two analyzes separately. In this way you could perform a procedure to validate the results of your analysis

Hello Lies,

As already suggested, no concerns on the first analysis. It is pretty common to conduct a CFA on any scales prior to modeling … usually along with reliability measures (as well as the fit indices).

If I understand your second SEM, you are conducting a model/regression analyses for various groups to see if they differ. In so, you may need to evaluate the measurement invariance of your scale(s). You are ensuring the multi-item scale has the same properties across different groups. Conceptually, it is a test on whether the individual questions that make up your scale, perform in a similar manner when measuring the construct in different groups. Are they perceived in a similar way and do they mean the same thing to members in each group? If so, the CFA for each group should have similar loadings and variance when compared to other groups. The test confirms whether this is so.

Some readings that may be helpful include:

Chen, F. F. (2007). Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464-504. doi:10.1080/10705510701301834

Cheung, G., & Rensvold, R. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Struct. Equ. Modeling, 9(2), 233-255.

Van De Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9(4), 486-492. doi:10.1080/17405629.2012.686740

Wishing you well in your research,

Tim

Following Giovanni,

you can implement a cross-validation strategy. Indeed, with multiple specifications of the model, there is the solid risk that the specified model and the following modifications may be driven exclusively by characteristics of the sample on which we tested the model (MacCallum et al. 1992). One strategy to addressing problems connected with multiple model specifications is the cross-validation analysis where a model is tested on a second independent sample (or more independent samples) drawn from the same population. You can split the total sample up in two independent randomly selected samples. First, you can test the hypothesised model on Sample A then the validity of the model is tested again on an independent sample – Sample B.

In

Bottoni, G., 2017. Validation of a social cohesion theoretical framework: a multiple group SEM strategy. Quality & Quantity, 1-22,

You can find all the steps and general procedure.

Best

Gianmaria

I have a sample size of 944.

My procedure now:

The results of the model fit of the CFA are different in the second SEM. What does this mean? How do I have to handle this?

Do I correctly interpret the given answers if I adept my procedure as follows:

I guess the results of the model fit of the CFA will be different as well in this second SEM, so same questions:

The results of the model fit of the CFA are different in the second SEM. What does this mean? How do I have to handle this?

Thanks so much for your help!

Hello Lies,

It seems like you are proceeding well, and have a decent sample size. Nicely done. Giovanni raises a good point. Are your factors from previous validated measures? If so, then an CFA on the whole dataset is fine - you are just confirming whether the validated measures run as expected with your participants. However, if these are factors you have devised, then you need to do a EFA (on half of the sample set, as Giovanni noted) and then a CFA (on the 2nd half to confirm). In all CFAs, you are looking for good fit statistics.

On your first question: Model fit statistics will almost always certainly differ b/w different models. That is not a problem. The issue you are looking for is whether the fit statistics are good/adequate. Generalising … if it fits the data, it fits. One model fitting ‘better’ than the other, does not really come into play (here I am talking about CFA fit stats like chi2, CFI, TLI, RMSEA … and NOT comparing nested models or AIC, BIC stats).

How you are using the clustering in your models is important. If you are simply using it as a predictor (IV), then ignore the following. However, if you are analysing a single model and hypothesising whether it differs between the clusters, then the following pertains - Before comparing the groups in one model, you should run them separately and ensure each group ‘fits’ the modelling. If that is the case, and they then pass the invariance test, then you are good to go on conducting a group analysis.

The splitting of the sample for your clustering analysis is unnecessary and not commonly done. It is almost universally conducted on a complete data set, saved to the dataset, and then analysed/modelled. This is important in your situation (opportune might be more appropriate) as you would lose power in your analysis, if your groups were roughly half the size (n).

Some references for power of analysis in SEM:

Hancock, G. R., & Freeman, M. J. (2001). Power and sample size for the root mean square error of approximation test of not close fit in structural equation modeling.(Statistical Data Included). Educational and Psychological Measurement, 61(5), 741.

Hancock, G. R., & French, B. F. (2013). Power analysis in covariance structure models. In G. R. Hancock & R. Mueller (Eds.), Structural equation modeling: A second course (2nd ed. ed.). Charlotte, NC: Information Age Publishing, Inc.

Muthén, L. K., & Muthén, B. O. (2002). How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 599-620. doi:10.1207/S15328007SEM0904_8

Wishing you well,

Tim

Dear Tim,

I'm so grateful for your help! It's the first time I'm conducting a SEM, so I have to learn a lot.

I am confirming a previously validated scale, so I'm only conducting a CFA.

The first SEM (with the full dataset) gives a model that fits (chi2, CFI, TLI, RMSEA are all 'good'). In the second SEM (grouped), the chi2 is higher, but so are the df making chi2/df lower than in the first SEM. The other fit stats are good, only the TLI has a value of .866. Is this a problem?

I'm using the clustering to create profiles. My research concerns job motivation and I want to create profiles of differently motivated groups. I then want to compare these groups on professional development. If I read your answer well, this is the first option and it doesn't concern 'analysing a single model and hypothesising whether it differs between the clusters'.

Taking this into account, I can follow the first procedure I mentioned?

Do you know by any chance a good example of an article reporting a (grouped) SEM?

Thanks so much,

Lies

Hello Lies,

Your approach and results are very encouraging … and everyone in SEM was a ‘first-timer’ at some point. I’ll throw a few suggestions your way, and you see what works. In order of your comments:

Glad to hear about validated scale; makes it all more defensible for CFA work. Good news.

It sounds like you are conducting a group analyses after devising your clusters; that is, do you have a variable in the second SEM (grouped) that states the cluster number (i.e. 1, 2, or 3) and are you using this in your model as the ‘grouping variable’? I may be misinterpreting. However, if this is the case then you ‘should’ model each group separately first for fit (the second option). Although some articles which are only comparing means, don’t necessarily test each group (see articles below). The first option would involve the ‘cluster variable’ being used as a predictor of other variables – in such a situation, you are controlling for the clusters, or more likely, trying to show that the clusters have an effect. If so, you have 3. Do they all differ?

On the TLI value of .866. That is low (see refs below) and is a concern. TLI is like CFI, but compensates for complexity of the model. So, we are looking for areas of misfit. I would suggest starting with your initial model and looking at modification indices and standardised residuals for areas of misfit. Be careful how you go about adding additional parameters, as they need to be defensible. Most factor analysis books have a section on this (e.g., Kline, 2016, p. 283; Brown, 2015, chapter 5 and p. 252). Then move on to your grouped SEM and check there.

Some examples of motivational cluster/group analysis papers are:

Watt, H. M. G., Richardson, P. W., & Wilkins, K. (2014). Profiles of professional engagement and career development aspirations among USA preservice teachers. International Journal of Educational Research, 65, 23-40. doi:10.1016/j.ijer.2013.09.008

Conley, A. M. (2012). Patterns of motivation beliefs: combining achievement goal and expectancy-value perspectives. Journal of Educational Psychology, 104(1), 32-47. doi:10.1037/a0026042

Howard, J. L., Gagne, M., Morin, A. J. S., & Forest, J. (in press). Using Bifactor Exploratory Structural Equation Modeling to Test for a Conuinuum Strucre of Motivaiton. Journal of Management.

TLI reference:

Hu, L., & Bentler, P. (1999). Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives. Struct. Equ. Modeling, 6(1), 1-55. doi:10.1080/10705519909540118

Marsh, H., Hau, K. T., & Grayson, D. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald (pp. 275-340). Mahwah NJ: Erlbaum.

Page 115-118 in - Little, T. D. (2013). Longitudinal structural equation modeling: New York : The Guilford Press.

Correcting for misfit:

Kline, R. B. (2016). Principles and practice of structural equation modeling (Fourth ed.). New York: The Guilford Press.

Brown, T. (2015). Confirmatory Factor Analysis for Applied Research (Second edition ed.). New York: Guilford Press.

Lots to digest. All the best and wishing you well on your SEM journey,

Tim