Hi all,
We are working on EFA/CFA analysis on a dataset.
We choose 'randomly select the cases' in SPSS 'Select Case' tab. Our rational is that we'll use the samples purely for factor analysis. We will not do any regression analysis. So, a simple action of randomly split the sample is appropriate.
We wonder if we should split the sample by controlling a variable, such as gender. We are not sure what difference it will make. Is this action necessary for straight-forward factor analysis?
Many thanks.
Hello,
I don't understand why you want to split the sample. If you split the sample according to gender and both genders have different factor structures, than the split will be appropriate as the model can adapt to the different structures. If both genders have the same structures (which is plausible in most cases) then splitting will unnecessarily reduce the sample size in each group with all negative consequences (increase of sampling variations of all relevant parameters.
Best,
Holger
Like Holger Steinmetz , I don't understand your rationale. In particular, it is unclear to me what you mean by "Our rational is that we'll use the samples purely for factor analysis. We will not do any regression analysis. So, a simple action of randomly split the sample is appropriate."
It is not clear to me what a random split has to do with whether or not you will run a regression analysis. In general, the idea of fitting EFA to one half of the sample and CFA to the other half has fairly limited utility. If you split the sample randomly, only sampling fluctuations (or a failed randomization) could account for any differences in the factor structure or parameter estimates between subsamples. It is thus not clear what you would learn from these analyses. In addition, using only half the sample limits your power for tests of significance (e.g., tests of model fit) and leads to less precision (higher standard errors) in the parameter estimates as compared to estimating the combined sample as a whole.
Thanks Holger Steinmetz Christian Geiser for sharing your views.
We managed to collect a large sample (>1000). Splitting the original dataset leaves us adequate participants for each subsample. Therefore, we made the decision to split the sample with the support from previous literature (Byrne, 2012; Brown, 2015; Kline, 2015; Wang & Wang, 2012). We wonder if you would kindly share some evidence supporting the disadvantages of splitting the sample. We probably need more reading on this topic. Many thanks.
-Brown, T. A. (2015). Confirmatory Factor Analysis for Applied Research (2nd ed.). New York: Guilford Publications.
-Byrne, B. M. (2012). Structural Equation Modeling with Mplus: Basic Concepts, Applications, and Programming. London: Routledge.
-Kline, R. B. (2015). Principles and Practice of Structural Equation Modeling (4th ed.). New York, NY: Guilford Publications.
-Wang, J., & Wang, X. (2012). Structural Equation Modeling: Applications Using Mplus. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781118356258
Hello Xiaoyi Shu
again, you did not answer why you want to split the sample. I can imagine that you want to adhere to the practice of cross-validation in the literature. This is this idea that when you change the model post-hoc and you do this in the "calibration sample" and a check in the "validation sample" brings the same results, then the post-hoc change is supported.
This idea is one of the many examples how predictive modeling and causal modeling (SEM, CFA) became intertwined during the decades. Cross-validation recently became very potent in the context of machine learning (=predictive modeling) where you apply very flexible functions that can fit the hell out of a feature space (i.e., the pattern of relationships among all variables). Thus, you can fit complex, nonlinear and interactive relationships. This flexibility comes at the price of overfitting (fitting the noise). In this scenario, cross-validation is very potent as the samples resulting from the split will not have exactly the permutations of all data points). Hence the overfitting will become visible and makes sense.
In the realm of SEM/CFA (but any factor model), relationships are linear and thus, not very prone to this kind of overfitting. The danger here is rather "fitting wrong structures" to the data (wrong parameters or factors). This will not change from sample to sample and hence, cross-validation will not provide evidence for the viability of your decisions. One can easily show by simulation that if you fit a wrongly specified model to the data (by tinkering) and you repeat the exercise in a splitted part of the sample, you will get the same seemingly supporting result.
Christian's and my note that you only eliminate valuable N by splitting does not need special references as it is the main implication of the role of N for the imprecision / uncertainty of any parameter estimation. Every introductory textbook on statistics will tell you that.
HTH
Holger
Holger Steinmetz Thanks for adding more explanation and insights. I have been trying to put your comments into our study context. Maybe I didn't make our question clear enough. I will probably need more time to digest your comments. Some of the points have given us more discussion of our study design. Thanks again.
- Badges
- Science topic
- Similar topics
- Statistics
- Survey