Chris - I was interested in your question "why is [SEM] being used in preference to other methods", and your recommendation that "authors would probably be better advised to use [PLS] rather than SEM".

Rönkkö and Evermann (2013) would strenuously disagree - and have articulated the reasons why in the following paper.

Rönkkö, Mikko and Joerg Evermann (2013), A Critical Examination of Common Beliefs About Partial Least Squares Path Modeling. Organizational Research Methods 16(3): 425-448.

It's a pretty harsh critique of the use of PLS for 'testing' theoretical models. Certainly, my advice to authors is now rather more biased in favor of covariance based SEM approaches (e.g., https://www.researchgate.net/post/Advice_on_SEM).

As for the original question - yes, you can use SEM with a convenience sample. You just cannot generalize from the sample to the population with known confidence from a convenience sample, no matter whether you use correlation, regression, SEM, PLS, or anything else. However, when testing theory, one is rarely interested in assessing some aspect of a sample and generalizing it to the population. Rather, one often first develops a theory, in which zero and no-zero relationships between all variables in a structural model are specified. Then one compares the theory to what is happening in a sample, recognizing that misfit can be interpreted as evidence that the theory is wrong in some way, and that fit is merely a failure to provide evidence that the theory is wrong (but is not evidence that the theory is correct).

Testing the theory on a conveniently chosen sample does not change this notion.

One can also think of the use of a convenient sample as being somewhat akin to traditional exploratory research. If the data in the sample are not supportive of the theory, then it may be that something unique about the sample is the cause of the model failure, perhaps because the theory is missing a moderator that takes on an extreme value (or extreme values) in the sample chosen (say). In which case, the researcher needs to reflect on what the possible variables might be that are missing from the model, or what the model should really look like. So - testing theory in unique samples (which might be conveniently chosen, but don't have to be) can help with theory modification. Much depends on the composition of the sample, and whether one is “lucky” in being able to isolate variables in it that could be candidates for entering into the theory.

Of course, convenient samples can hide real relationships too, and so are somewhat dangerous (might lead to rejection of the theory when it is true). For instance, it may be that a convenience sample will suffer from range restriction problems (and so mask relationships that exist). Consequently, as with any exploratory research, one should supplement convenience sampling with more probabilistic approaches too.

Finally, as for using the same data for undertaking EFA and CFA, I think the main criticism is that one is capitalizing on chance. That is, there is a good chance that the EFA will pick the “best set of items” unique to the data set, and the CFA on the same data set will confirm that the items are good. But on a second independent data set, the best items might differ (relative to the first data set). So conducting the EFA and the CFA on difference data sets makes the task of validating the measures that much harder (and so more rigorous). See the second article Amir pointed to...

Best regards

John

Hi Amir

I agree with your general point - and indeed we ask for many things in our author's guidelines that other journals do not seem to do so (e.g. copy of questionnaire, sets of descriptive stats including those of dispersion, AVE, CRs etc.) - and yes SPSS for one provides now more tests for missing data. Unfortunately it seems many writers do not read author's guidelines.

If SEM is not a case of casual modeling - as many still seem to think it is, then of course one can begin to ask - why is it being used in preference to other methods. There is of course PLS as a more 'tolerant' technique - and I think authors would probably be better advised to use this rather than SEM.

One related issue - the 'purists' would argue that no one sample should be used for both EFA and CFA - any comments?

Cheers

Chris

I can see no problem in that. The most important thing is to make sure that the sample is adequate by Kaiser-Meyer-Olkin and Barlett`s test of sphericity measures, when you use factor analysis.

Some traditionalists would argue that EFA and CFA are premised on different functions and the same set of data cannot be simultaneously used for exploration and confirmation.

Chris - I was interested in your question "why is [SEM] being used in preference to other methods", and your recommendation that "authors would probably be better advised to use [PLS] rather than SEM".

Rönkkö and Evermann (2013) would strenuously disagree - and have articulated the reasons why in the following paper.

Rönkkö, Mikko and Joerg Evermann (2013), A Critical Examination of Common Beliefs About Partial Least Squares Path Modeling. Organizational Research Methods 16(3): 425-448.

It's a pretty harsh critique of the use of PLS for 'testing' theoretical models. Certainly, my advice to authors is now rather more biased in favor of covariance based SEM approaches (e.g., https://www.researchgate.net/post/Advice_on_SEM).

As for the original question - yes, you can use SEM with a convenience sample. You just cannot generalize from the sample to the population with known confidence from a convenience sample, no matter whether you use correlation, regression, SEM, PLS, or anything else. However, when testing theory, one is rarely interested in assessing some aspect of a sample and generalizing it to the population. Rather, one often first develops a theory, in which zero and no-zero relationships between all variables in a structural model are specified. Then one compares the theory to what is happening in a sample, recognizing that misfit can be interpreted as evidence that the theory is wrong in some way, and that fit is merely a failure to provide evidence that the theory is wrong (but is not evidence that the theory is correct).

Testing the theory on a conveniently chosen sample does not change this notion.

One can also think of the use of a convenient sample as being somewhat akin to traditional exploratory research. If the data in the sample are not supportive of the theory, then it may be that something unique about the sample is the cause of the model failure, perhaps because the theory is missing a moderator that takes on an extreme value (or extreme values) in the sample chosen (say). In which case, the researcher needs to reflect on what the possible variables might be that are missing from the model, or what the model should really look like. So - testing theory in unique samples (which might be conveniently chosen, but don't have to be) can help with theory modification. Much depends on the composition of the sample, and whether one is “lucky” in being able to isolate variables in it that could be candidates for entering into the theory.

Of course, convenient samples can hide real relationships too, and so are somewhat dangerous (might lead to rejection of the theory when it is true). For instance, it may be that a convenience sample will suffer from range restriction problems (and so mask relationships that exist). Consequently, as with any exploratory research, one should supplement convenience sampling with more probabilistic approaches too.

Finally, as for using the same data for undertaking EFA and CFA, I think the main criticism is that one is capitalizing on chance. That is, there is a good chance that the EFA will pick the “best set of items” unique to the data set, and the CFA on the same data set will confirm that the items are good. But on a second independent data set, the best items might differ (relative to the first data set). So conducting the EFA and the CFA on difference data sets makes the task of validating the measures that much harder (and so more rigorous). See the second article Amir pointed to...

Best regards

John

Hello!

I think PLS-SEM is a great tool if used wisely.

For the recent critics on PLS (by Rönkkö and Evermann),see this article that aims to response to those critics:

https://www.researchgate.net/publication/260037058_Common_beliefs_and_reality_about_PLS_Comments_on_Rnkk__Evermann_(2013)

Article Common Beliefs and Reality About PLS: Comments on Rönkkö and...

Dear Amir,

You say that "the days of regarding SEM models as "causal models" are hopefully long gone" - however, I could not disagree more. To the opposite! :) The last decade have been defined by a unimaginable cowardness towards causal thinking leading to transforming SEMs in a simple descriptive tool that has no causal implications. Instead of conceptualizing latent variables as reflections of truly existing phenomena, we talk vaguely about "constructs" and instead of causal effects we talk about "relationship".

However, the content of an SEM unavoidably IS causal. If the model is causally misspecified, the parameters are not even descriptively useful and in the extreme could mean nothing at all. However (and I could guess that is your point), one should not naively interprete any output as causal. There you're absolutelty right.

But what we can do, is to strengthen the causal evidence of the model by:

a) clearly conceptualizing "constructs" (in terms of empirically existing entities with supposed causal functions)

b) Importing essential causal restrictions (e.g., by using instruments) that imply burdens that the model has to pass if it is correct

c) rigidly test these restrictions (by using the chi-square test).

The fact that you cite the Bollen/Pearl chapter is rather funny as this chapter is a plea for causal modeling and not a critique! Especially Judea Pearl at present touts as loud as nobody else for clarifying and thus strengthening the causal content of SEM.

Hence, in paraphrising your statment, the days of lax SEMing with ontologically unclear variables, causally undefined and unclear parameters and lax evaluation of model fit are over :)

I am aware that the topic of this thread complicates things a bit - but as John said in the beginning, all problems with data (e.g, outliers, range ristriction, distributions) or the sampling process does not decrease the value of testing the model and learning from it.

Best regards :)

Holger