As far as I understand, you have data from multiple raters on the same construct(s). If this is true, then you could look into CFA models for multirater (or, more generally, multitrait-multimethod [MTMM]) data. There is a distinction between CFA-MTMM models for interchangeable raters (e.g., multiple randomly selected students evaluating their teachers) versus models for structurally different raters (e.g., therapist rating vs. patient self-reports). Models for randomly selected (interchangeable) raters can be analyzed using multilevel CFA.

Below are two references that you can start with:

Eid, M., Geiser, C., & Koch, T. (2016). Measuring method effects: From traditional to design-oriented approaches. Current Directions in Psychological Science, 25, 275-280.

Eid, M., Nussbeck, F., Geiser, C., Cole, D., Gollwitzer, M. & Lischetzke, T. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253.

Dear Christian, Thank you very much for your response and for the suggested articles. Based on the suggested readings, the most appropriate analysis seems to be the one based on the Correlated Trait-Correlated Uniqueness (CT-CU) model. However, I will try to explain myself better to understand if the application of this model would be correct. I have an instrument composed of 10 items/dimensions. Each item/dimension is evaluated on a scale from 1 to 10. This value is decided together by the patient and therapist, meaning: the patient and therapist read the item, discuss it together, and come to an evaluation (the most appropriate for both) of the item/dimension; deciding together the rating to give (for example, 3). The same is done for the other 9 items. This is done for - for example - 100 pairs of patient-therapist (with multiple therapists; but a therapist could be the same for more than one patient). My goal is to evaluate the internal consistency and/or the factorial structure of this instrument, with a method that goes beyond the calculation of the ICC index. I would like to capture (or not) the internal consistency taking into account the influence that the fact of arriving at a common evaluation for each item for 100 different pairs/items might have. I thought of CFA as a method to analyze at least one latent unique factor (even though the items are not so similar, so I don’t necessarily expect a single latent factor), because the instrument, at least qualitatively, has already been structured into 10 items/dimensions. I thought about mixed models given their ability to take into account individual differences (e.g., subject level) which seems very present in the type of validation I would like to do (I mean the influence of individual differences).

Thank you

Dear Salvatore

Do you have separate ratings from the therapist and patient before their discussion and common rating? In that case you would also have the possibility of comparing their individual evaluations apart from the consensus rating.

Such data would be of the type dyadic data with distinguisable members, since you have pairs of therapist and a patient (examples of dyades with indistinguisable members could be two rommmates or two collegues evaluating each other)

A good reference for the possibilities for studying such data is Kenny, Kashy and Cook (2006) Dyadic Data Analysis. New York: Guilford.

A short treatment introducing a combination of SEM and multilevel modeling is

Newsom, J.T. (2002). A multilevel structurual equation model for dyadic data. Structural Equation Modeling, 9(3), 431-447

Thank you Jan for you answer and for the papers. Dyadid data semms to be a solution. However, In this case I'm not in the situation were each one evaluate each other, but both derive a unique evaluation for each item (two persone talk togheter and decide the item score). Is there a type of dyadic data for this situation? Could, in your opinion, a RIIFA model be a solution? Or should I consider this as a "normal" case for a CFA?

Just to clarify: My comment did not presuppose the two responders evaluate each other, just situations where two persons evaluate the same issue, like two persons responding to the same items.

I am not aware if you already have gathered the data or if you are in the planning phase of the data gathering. In the latter case, I would highlly recomment to also register the initial individual ratings of the therapist and the patients, as this gives you valuable information.

As for the data processing of the consensus rsæponses, I will refer to the earlier response of Christian Geiser.

Tow potential solutions: MNLFA and multilevel CFA

Moderated non-linear factor analysis by Dan Bauer. He and Andrea Hassong have a 2009 paper in psych methods and Dan has a solo paper in psych methods in 2017

Bauer, D. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077

Curran, P. J., McGinley, J. S., Bauer, D. J., Hussong, A. M., Burns, A., Chassin, L., ... & Zucker, R. (2014). A moderated nonlinear factor model for the development of commensurate measures in integrative data analysis. Multivariate behavioral research, 49(3), 214-231.

You may also consider multilevel confirmatory factor analysis: Kris Preacher and Michael Zyphur have done a lot of really good work.