Hi
I have two groups (males = 420 and females = 763) which have taken an instrument we have translated into a new language. We are attempting to validate the instrument. Overall, we found modest fit during CFA for the combined group (likely due, in part, to 2 of the 32 items having below .3 loadings - the rest were great). Those two items remained poorly loaded across multiple CFA iterations (single latent, correlated oblique, and bi-factor), but were significant. My co-author and I are in disagreement about if those items should be dropped and the model re-run. It is my belief that doing so would provide both the basic instrument fit but also the fit as we would recommend it be interpreted in the new translated version as such poorly loaded items offer very little interpretive information. In general, the fit is similar to that observed previously.
When we conduct invariance testing it fails immediately, and badly (results pasted below). My co-author and I are also in a disagreement about how we can interpret this. It is my understanding that the unequal sample sizes will not cause us to be more likely to find between group variance and that, in fact, it makes it more likely for us to fail to reject the null (e.g., that the groups are invariant). Simulation studies I have seen (Yoon & Lai, 2017), and other references to sample size considerations, seem to support this. My co-author remains concerned that our sample size has caused this and that we cannot interpret our results as indicating non-invariance for males and females.
Two questions
====
1. Anyone know a reference to a test development book that says its ok to drop poorly loaded items during instrument translation?
2. Any thoughts, references, or interpretations of the measurement invariance issue as it relates to sample size differences?
Observed Configural Results
====
Chi-Square Test of Model Fit
Value 15611.280*
Degrees of Freedom 460
P-Value 0.0000
Chi-Square Contribution and P-Value From Each Group (degrees of freedom = 230)
MALE 3825.047 0.000
FEMALE 11786.233 0.000
RMSEA (Root Mean Square Error Of Approximation)
Estimate 0.236
90 Percent C.I. 0.233 - 0.239
Probability RMSEA
With regard to your first question, after translating the instrument, have you assessed the face validity and performed pilot testing?
With regard to second question, in my personal experience as data analyst, your sample size of both groups is enough to gauge the variance between the groups. I will suggest you perform measurement model invariance tests like Configural test at first level, if it is OK then Metric test at second level if it is OK then Scalar test at level 3. If you fail at any level, look what are the reasons. Your above given fit indices are not meeting the said threshold criteria. I think rather to focus sample size, you guys should divert your attention toward the quality of responses.
Hi Paul,
Great question-
It is generally not a problem, unless samples are drastically different (>2x). In any case, I would recommend looking at Meade and colleagues' work, for example, they offer a calculator for "practical effect sizes" based on observed differences (accounts for sample size)
Hi Paul B Ingram ,
For your first question, you may refer to Translating and Adapting Tests for Cross Cultural by Ronald K. Hambleton and April L. Zenisky (https://books.google.com.my/books?id=6VpQNlc9pUAC&pg=PA46&output=html_text&source=gbs_toc_r&cad=4).
Hi Paul,
I found it interesting.
The sample size is quite adeqaute to undertake the test of invariance between two groups. It's independent of translation to other language which i guess gone through the content validity before the survey started off.
Loading of 0.3 you may keep it for time being as loading itself doesn't decide the dropping of items but lower loading may cause model fit issue. From the results i found the issue with chi-sqaure fit and other fit indices though RMSEA is below 0.08. My suggestion would be to undertake CFA for each gp separately and check if the model fits. If the p-value is less than .05 then go for Boolenstine p-value within AMOS-SEM.
Drop items here looking at the square -multiple correlations for each item if less than 0.3. Also check the regression coefficients and their significance level whcih allows you drop the insignificant items.
All these steps in CFA will clean the items with good fit of the model. Then you an move on to path model to perform the invariance test between two gps.
Thank you all for the answers and resources. They aligned well with my understanding of the literature.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size