Dear David,

I dont think merging items generated from 2 completely different source/method is a correct option for one single research. However, you can run validity testing (cronbach etc.) on your merged questionnaire to test. However, its not recommended.

Praheli Dhar Chowdhuri Thanks for your input. I agree, but I'm looking for references I can cite to justify that decision.

An important issue here is the extent to which items were previously validated to reflect a theoretical construct (factor).

I am not familiar with the SILL, but it sounds like that is a validated scale. It would make little sense to perform EFA on (just) that because the factorial structure is already known.

So, I would suggest that an EFA on all items simultaneously would make most sense.

Heiko Breitsohl Yes, the SILL has previously been validated in many contexts. That's kind of my point. By doing an EFA on just the SILL items that I used (I did not use the entire SILL), I'm looking for latent factors, not confirmation of the already validated factors. I do not want those latent factors interfering with the factors in the other survey questions.

Additionally, the other survey questions were all generated by Taiwanese university students. The SILL data came from many many international students (studying ESL in America, I believe) almost 30 years ago. These are obviously extremely distinct groups. Is that not validation enough to explore possible factors in those instruments separately?

Also, while I may also look at both scales together at some point, I would first like to explore them separately and then, examining those factors from each, perhaps combine them to see if those factors hold up in a combined scale. That also seems to be a "common sense" approach, but maybe I'm wrong.

Hii @David ....there are few suggestions that I want to incorporate regarding your question:

Firstly, why have you merged both the scales? What is the motive behind doing so..? Are they measuring a same/similar construct/concept? Is your work related to some kind of scale development?? Actually, these are the questions which may arise during review...You just need to be clear with your motive regarding merging two different scales or so (i.e., one is the standard and the other one which you have developed through qualitative study).

Secondly, if your work is about developing a new scale ...then I would suggest using both scales (ie., your developed scale and standard scale) in a small sample (~200 respondents) for a pilot study. This will help you establish initial validity of your scale through EFA (i.e., getting an idea about the factors within the scale; items loading within a same factor indicates convergence and items differently loading on different factors represents divergence, any cross loading must be discarded). After this, you can correlate the scores for your developed scale and the standard scale. If ,in case, they are measuring a similar theoretical construct/concept, they must be corrected adequately. An adequate correlation will show that the concurrent validity of your developed scale is established.

Finally, after establishing initial validity of your scale through EFA, you can use a larger sample to determine the psychometric properties of your developed scale through CFA.

Now to your question:

Perform EFA on 20 items that you have captured from qualitative study. In doing so, you will get a refined picture of what items in the factors of you scale should consist.

Performing an EFA on already an established scale only makes sense when the scale has not been previously established in your context of study/or any similar context.

But anyway, after performing EFA you need to go for CFA which ideally should be performed on a different and larger sample.

Hope these points help...!!

Reference:

Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967-988.

Hair, J., Black, W., Babin, B., Anderson, R., & Tatham, R. (2010). Multivariate data

analysis (7th ed.). Upper Saddle River, NJ: Pearson Prentice Hall.

Furr, R.M & Bacharach, V.R. (2013). Psychometrics: An introduction (2nd ed.).

Thousand Oaks, CA: Sage.

Vey Sorry. I do not work anymore with statiscs. Best regards