Imagine we have ten pictures of faces and we have them rated for attractiveness. Let's say 20 participants rate them using a 1-5 Likert scale (1=unattractive; 5=attractive). Another 20 participants rate the same faces using a 1-7 Likert scale (1=unattractive; 7=attractive). Finally, another 20 participants rate those same faces using a 1-9 Likert scale (1=unattractive; 9=attractive).
We assume there is some underlying quality, "attractiveness", that these different scales are trying to quantify. How best can we analyse these data to get at the question of this underlying measure, how well each scale measures it, and whether some scales do a better job than others, etc.?
Thanks in advance for any ideas/suggestions!
If you have other measures of related constructs that were assessed in the same individuals, you could look at convergent validity (i.e., whether there are high correlations of the attractiveness scores with the other theoretically closely related constructs). Likewise, you could study discriminant validity relative to other, theoretically unrelated (or less strongly related) constructs. You could also examine at whether your attractiveness scores allow you to predict relevant outcome variables (criterion validity).
Just from an empirical point of view, you would need to first how good the inter rated relaiability is with each scale, for the same set of stimuli. Once you have established that each individual scale is reliable, then you would look at the strength of association between the scales, presumable using correlation. But you might also want to look for location and range effects across different Likert scales, that might be easiest to do be recording each scale so that the middle point is zero.
To analyze these data, one approach would be to use multi-level modeling (MLM) or item response theory (IRT). These methods allow for the estimation of latent variables that underlie the observed responses, such as the underlying "attractiveness" in this case. Additionally, they can also estimate the measurement precision of each scale, and how well each scale is able to accurately measure the latent variable.
First, you would need to combine the data from all three scales and the ratings of all the faces together into one dataset. Then you would specify a multilevel model with faces as the first level and the participants as the second level. This will allow you to estimate the average attractiveness for each face, as well as the variability of attractiveness among the faces.
Next, you would include the Likert scale used (1-5, 1-7, or 1-9) as a predictor in the model, along with the rating of each face (1-5, 1-7, or 1-9) as the outcome variable. This would allow you to estimate the effect of the Likert scale on the ratings of the faces, and how well each scale is able to measure the underlying "attractiveness" construct.
Another approach could be to use IRT models such as Rasch or Two-Parameter Logistic (2PL) model, which will also allow you to estimate how well each scale measures the underlying construct of attractiveness, and how well they are able to differentiate among the faces.
It's worth noting that, regardless of the method you choose, it's always important to check the assumptions of the model and the data for normality, outliers, and missing values before analyzing the data. It's also recommended to consult with a statistician or researcher experienced in these methods for guidance.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire