I am not certain this is what you are requesting, but A popular index of inter-rater reliability , inter-rater agreement , or concordance , is Cohen's kappa (Cohen1960).

kappa= [ P( observed_agreement ) - P( hypothesized_agreement )] /

[100%-P( hypothesized_disagreement ) ].

Of course, if you hypothesize the judges are the same,

P( hypothesized_disagreement) =0.5 (50%). Kappa seems to be more or less continuous, so for factors, you may have to use some sort of binning.

Cohen, J, 1960. "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20 (1): 37–46.

I hope this helps.

Evening, Pedro,

As long as you can theoretically derive a construct from the items you have, you can do it.

I will tell you the procedure for SPSS and AMOS.

In SPSS you go to analyze - dimension reduction. Here you select the method - if you use PLS I would go for Principal components analysis. (Although, some would argue it is not correct). You select a rotation varimax, promax or oblimin (it depends on the theory behind your items, e.g. are they influencing each other or rather independent). Then you select scores and save as regression. (Some argue that Bartlet score is more correct). Voila - new factors appear in you dataset.

The procedure is close to EFA. In the table with a pattern matrix you will find the loadings.

In AMOS you have to create a model with your constructs and subordinated items. Select 'standarfused values' for output analysis. You ran the model and go to the results. You can find there the model fit as well as standardized regression weights. These are your loadings. Please be aware that Amos uses maximum likelihood as the algorithm by default.

There are several tutorials on YouTube on this issue.

Best,

Eugene

Hi, Lewis. Thanks for your reply, but I think that sadly it won't help me much. I'll give you more details. I want to use judges evaluation of social desirability as a fixated factor loading to items responses in a SEM. I'll probably work with a bifactorial SEM controlling social desirability by fixating items factor loading in F1 and estimating the factorial loading for items responses in F2. For F2 I'll estimate the factor loading using the regular response of participants for the measurement scale. My doubt is about the convertion of judges evaluation to an index I could apply in SEM for fixating factor loading of F1.

Thank you, Yevgen Bogodistov, but I still looking for an alternative procedure. If we use PCA it may integrate in the model "irrelevant variance".I'm not really sure if it's possible in an alternative approach, but lets see what kind of ideas we may have about it in future contributions. Anyway, I'm thankful that you shared your knowledge to contribute with me. Best regards!

The answer is no, due to factor indeterminacy. The correct or true case values for an indeterminate common factor are not unique--an infinite number of different sets of case values exist, all of them perfectly consistent with the factor model (Grice 2011 is a readable treatment). Moreover, the correlation between a common factor and any variable outside the factor model is blurred by factor indeterminacy--we can only determine such a correlation within a certain range, which is broader as the factor is more indeterminate (Steiger 1979). That is the problem you would face if you try to construct this social desirability index in one model and then use scores from it in another model.

You certainly could assign some score to each participant based on some index of their other scores. You might even have *other* people judge each judge's scores and assign each judge a grade. For that matter, you might treat this as multilevel data, with a given judge's scores clustered by judge. Then you could in principle distinguish second level judge effects from first level item judged effects.

Grice, J. W. (2001). Computing and evaluating factor scores. Psychological Methods, 6(4), 430-450. doi: 10.1037/1082-989X.6.4.430

Steiger, J. H. (1979a). The relationship between external variables and common factors. Psychometrika, 44(1), 93-97. doi: 10.1007/BF02293788

Thank you, Prof. Edward Rigdon. If it's possible, could you check this formula?

Rb = s

It had been propposed by Pettersson et al. (2014) as a possible solution, but once you bring the indeterminancy as a problem, I'm not sure if it would be applied.

I'm interested in understand if the factor indeterminancy also applies to judges evaluation if I create a factorial loading from it.

Best Regards,

Pettersson, E., Mendle, J., Turkheimer, E., Horn, E. E., Ford, D. C., Simms, L. J., & Clark, L. A. (2014). Do maladaptive behaviors exist at one or both ends of personality traits?. Psychological Assessment, 26(2), 433-446. doi: 10.1037/a0035587