It seems to me:

1. If you have adapted the items in a scale to a different context, that means it is now intended to measure a different construct. (Similar to the original construct, and maybe even closely related, but not the same.) The items themselves have presumably been reworded, even if only slightly, to correspond to the new context. Therefore reliability, validity, etc. need to be re-established.

2. Your preliminary results seem to say the items do not relate (to each other, and to the construct) in exactly the same way in both groups (studies). Multi-group confirmatory factor analysis will quantify those differences in the measurement model. It can also help you "control for" those differences when comparing the relationships of the constructs between the groups, if you estimate the entire model (measurement and substantive segments) all at once. However, if the between-group differences in the measurement model are large, it does call into question whether the constructs represented by the latent variables in the substantive model are actually the same.

3. If you don't or can't estimate the entire model all at once with multi-group DFA, a practical work-around could be to combine the two groups and go through the usual steps for scale construction (examining EFA, alpha, item-to-total correlations, etc.) to determine which subset of the items produces the best scale for the combined groups. Use those and only those items to create the scale. Then divide the data back into your two original groups, and proceed with your comparison of the groups based on that scale.

4. Another work-around would be to form the scale from only those items that "work" in both groups: i1, i3, and i4 in your example. Or, as others have suggested, keep all 6 of the original items. Check to see if either of those approaches yields acceptable alpha values in both groups.

4. I would not under any circumstances use two different subsets of items to form the scale in the two groups. If you do that, any comparison you make is an "apples to oranges" comparison. If you find a difference between groups, you can't know whether the groups actually differ on a particular construct, or the constructs measured by your two different subsets are really two different constructs.

I would not rely as much on statistical significance of the factor loadings, but on the invariance of the factor loading between groups.

By the way, you could treat your studies as separate groups as well and compare them directly.

With established scales, people generally do not do analyses for each sample and then use the sample-specific results to decide which items to drop. They usually just use all the items. One possible exception is if the scale is developed in one country and then used (perhaps translated) in another. In that case you would want to be sure the items are working in the new setting. Even in that case, I would not create sample-specific versions. I would decide which items to use and use them in both samples.

Multigroup Confirmatory Factor Analysis (CTT) or DIF (IRT).

Thank you Heiko Breitsohl , Paul E. Spector and Matthew Kerry .

Now, I have supplementary questions for Heiko and Paul.

Heiko Breitsohl, first, thanks for pointing out the in-variance of factor loading between the groups. I read this somewhere but forgot. Thanks for reminding. 2nd, if there is significant invarience of factor loading between the groups, does it mean we cannot make a comparison between two groups, like multi-group analysis?[Q1]

Paul E. Spector , my area is users' cybersecurity behavior, which lacks established scales. Most of the research is based on constructs which were developed by adapting it from other knowledge areas. For example, Attitude is a construct that is context dependent. A researcher developed a construct of Attitude towards system security, and thereafter, the same construct has been adapted by others to measure attitude towards different dimension of cybersecurity: web security, data security, email security, or even information security in general etc. So, I am not sure if such constructs are qualified as established scale (or am I wrong?). [Q2]

2ndly, when the context (dimensions) is changed, don't you think we need to run CFA or similar other tests to test reliability and validity of the construct? By the way, I am using SmartPLS- SEM.[Q3]

Coming to latter part of your comment that we should use the same items in both the samples. I have done the same. I have used same constructs to collect data from two respondent groups from two different countries. Data collection was carried out in same language. While examining the samples separately, I found that for few constructs, different sets of items are loading (such as the one I mentioned in the question). So the construct of Attitude is measured through one set of items in one population and different set of items in 2nd population. So can I compare Attitude of two groups while the construct is measured with different set of items? [Q4]

Thanks,

Ali

It seems to me:

1. If you have adapted the items in a scale to a different context, that means it is now intended to measure a different construct. (Similar to the original construct, and maybe even closely related, but not the same.) The items themselves have presumably been reworded, even if only slightly, to correspond to the new context. Therefore reliability, validity, etc. need to be re-established.

2. Your preliminary results seem to say the items do not relate (to each other, and to the construct) in exactly the same way in both groups (studies). Multi-group confirmatory factor analysis will quantify those differences in the measurement model. It can also help you "control for" those differences when comparing the relationships of the constructs between the groups, if you estimate the entire model (measurement and substantive segments) all at once. However, if the between-group differences in the measurement model are large, it does call into question whether the constructs represented by the latent variables in the substantive model are actually the same.

3. If you don't or can't estimate the entire model all at once with multi-group DFA, a practical work-around could be to combine the two groups and go through the usual steps for scale construction (examining EFA, alpha, item-to-total correlations, etc.) to determine which subset of the items produces the best scale for the combined groups. Use those and only those items to create the scale. Then divide the data back into your two original groups, and proceed with your comparison of the groups based on that scale.

4. Another work-around would be to form the scale from only those items that "work" in both groups: i1, i3, and i4 in your example. Or, as others have suggested, keep all 6 of the original items. Check to see if either of those approaches yields acceptable alpha values in both groups.

4. I would not under any circumstances use two different subsets of items to form the scale in the two groups. If you do that, any comparison you make is an "apples to oranges" comparison. If you find a difference between groups, you can't know whether the groups actually differ on a particular construct, or the constructs measured by your two different subsets are really two different constructs.

Thank you, Burke. Your response is comprehensive and answers all the questions (even subsequent ones) I had.

Much appreciated!

Burke D. Grandjean Thanks for the detailed answer. I have some doubts if you could please address.

What if I have reworded the same questionnaire in two languages and all the constructs remain otherwise same. Suppose, my two group analysis is based on gender and the questionnaires are in Spanish and German. Am I still supposed to establish reliability and validity for German and Spanish version of the instrument individually?

Regarding your second point, Let's say I am checking work stress between men and women in Germany and Spain. If I get really large between group differences in the results across contexts, would it still call for an inquiry whether it is measuring the same construct? Or there could be actually huge difference due to the context?

It sounds like you are using exactly the same set of items to form a scale in both Spanish and German, and that the items themselves are nearly identical (except for different languages and idioms). Nevertheless, I do think you should do a basic reliability check separately for each language version. Fundamentally, you have different questionnaires, applied to different populations. It's possible that an item or two got mistranslated, or that there's just no way to express the same thought exactly in the different languages. A quick check of Cronbach's alpha, alpha-if-item-deleted, and corrected item-to-total correlations would be worthwhile. It may tell you there's nothing to worry about, or it might say that deleting one particular item (in both languages) greatly improves the psychometrics of the scale.

If diagnostics like that leave you with a scale that has decent reliability, the same set of items, and the same number of items, in both languages, then you are probably safe to interpret the scale as measuring the same construct in both languages. That is especially true if the original questionnaire (in English or whatever) that you translated from was well validated by the previous researchers. If you get a large and significant gender effect in both languages, you can then say that men and women differ on that construct in both populations.

BTW, you seem to be using the term "context" to refer to the language or national differences. I think, in the original question as clarified in a second posting, "context" referred to specifics of the research problem -- taking questionnaire items that were designed to measure "system security" as one context and modifying the wording to apply to a different context like "email security." It does not sound like that is what you are dealing with.