Sam Crowley I don't see how removing items with poor factor loadings would lead to overfitting as doing so would remove (rather than add) free parameters from the model. We are typically more concerned about overfitting and cross-validation when we add additional model parameters in a data-driven way (e.g., error covariances between items, cross-loadings on other factors etc. based on model modification indices or model residuals).

I would recommend thinking about why these items don't load well on the intended factor in your study. For example, could this be due to you using a different population than the validation study (e.g., age or cultural differences, higher homogeneity of the individuals with respect to the construct(s))? Did your respondents not understand the question(s)/item(s)? Are the items less reliable or valid in your sample for other reasons? There must be a reason why the item(s) in question do not measure your factor(s) in the expected way.

In summary, I don't think it is necessarily bad practice to remove bad items. However, it would be good to understand why the items are bad measures in your study when they were good measures in previous studies.

Dear Sam,

Were the poor factor loadings from the scale developer's study or a new study? If they were poor loadings, why were those items retained in the first place? Were they retained in order to bolster the reliability of the instrument even if only slightly. As general rule of thumb, you should never change an instrument without re-validating with a sample from the population where you want to use it. Instruments are not valid in general, but for a specified population.

Good luck,

J. McLean

Dear sam

instruments (I mean instruments such as questionnaires, tests, sometimes checklists, and transcripts of interviews that have specific items and separate dimensions) are cultural, social, religious, or even gender-related. In general, whenever a tool is built under certain conditions, the validity of its appearance, the validity of its content, and the validity of its structure will be checked for reuse in different circumstances. In my opinion, in order to avoid deleting items and to better fit the data with the model in new conditions, it is necessary to check the content validity of the tool in the new condition.

Good luck

Hey Sam, you have received some great advice already. My tuppence is that I am always reluctant to remove items from a previously published/validated scale as this can change the possible range of scores, and the summed score, that can be derived from the 'new' scale. This makes it very difficult to compare descriptive statistics with other studies (unless the scale is scored by taking the mean of items, rather then the more common method of summing all the items). Removing items also has the potential of narrowing the nature of the construct being measured. I can live with pretty lousy items in order to avoid these problems.

Best wishes

Mark

What do you plan to do with the resulting scores? Will you be comparing them with other studies using the scales, looking for differences across your experimental conditions, etc.?

I think you are right.