Hello Christof,

For reliability, I'd suggest you have a look at McDonald's omega method. Here's a link that walks through a variety of packages for its estimation:

https://www.tandfonline.com/doi/abs/10.1080/19312458.2020.1718629?journalCode=hcms20

I can think of a couple of conceptualizations that might make sense.

1. SEM model, with full set of latent variables (all six dimensions), and 12 indicators (3 each for criterion 1, criterion 2...) for each proposed dimension. Note that this might result in the need for higher-order factors to capture criterion, a very busy model, or a very messy model.

2. An adaptation of Multitrait-multimethod approach (by Campbell & Fiske) in which criterion is treated as a method variable.

Good luck with your work.

Thank you very much for your answer! Oh, I didn't think of MTMM! That makes perfect sense :)

Cheers!

Assessing the reliability of a multi-criteria questionnaire involves evaluating its internal consistency and stability. Reliability ensures that the questionnaire consistently measures what it intends to measure, and the results are dependable and consistent over time. Here are some commonly used methods to assess the reliability of a multi-criteria questionnaire:

Cronbach's Alpha: Cronbach's alpha is a widely used measure of internal consistency. It assesses the extent to which all the items in the questionnaire are measuring the same underlying construct. The value of Cronbach's alpha ranges from 0 to 1, with higher values indicating better internal consistency. Generally, a Cronbach's alpha value above 0.7 is considered acceptable.

Test-Retest Reliability: Test-retest reliability assesses the stability of the questionnaire's scores over time. It involves administering the same questionnaire to the same respondents on two separate occasions with a time gap in between. The scores obtained from both administrations are then correlated. A high correlation coefficient indicates good stability and reliability.

Split-Half Reliability: Split-half reliability involves dividing the questionnaire into two halves (e.g., odd-numbered items vs. even-numbered items) and comparing the scores from both halves. The correlation between the two halves measures the internal consistency of the questionnaire. This method can be useful when the questionnaire is lengthy, but it may not be appropriate for shorter questionnaires.

Inter-Rater Reliability: If the multi-criteria questionnaire involves multiple raters or evaluators, inter-rater reliability can be used to assess the consistency of their ratings. It measures the level of agreement between different raters using methods such as Cohen's kappa or intraclass correlation coefficient (ICC).

Factor Analysis: Factor analysis can be used to examine the underlying structure of the questionnaire and identify the number of distinct factors or dimensions it measures. This method helps in assessing the construct validity and can indirectly provide insights into the reliability of the questionnaire.

Item Analysis: Analyzing individual items within the questionnaire can provide valuable information about their reliability. Items with low correlations with other items or the total score may need to be revised or removed to enhance the questionnaire's overall reliability.

Parallel Forms Reliability: If you have multiple versions of the multi-criteria questionnaire, parallel forms reliability can be used to assess the consistency between different forms. It involves administering both forms to the same group of participants and comparing their scores.

When assessing reliability, it's essential to consider the context and purpose of the questionnaire and choose the most appropriate method(s) accordingly. Multiple methods can be used to complement each other and provide a more comprehensive assessment of reliability. Additionally, it's good practice to pilot test the questionnaire with a small sample before conducting the main study to identify any potential issues with reliability.