I would like to construct a ten-item scale on the subjective importance of various goals in higher education (e.g., equal opportunities, gender issues, promoting sustainability etc.). When using rating (Likert) scales, many respondents will rate almost all of the items highly (response style, social desirability etc.). Therefore, a colleague suggested ranking in order to avoid the problem. My concern about rankings, however, is that they are ipsative by their nature AND ordinal with respect to their scale level. As a result, only very simple statistical methods would remain for data analysis.
Can anyone suggest how to deal with the problem? E.g., using ratings and given an instruction to make use of the full scale of possible ratings, from lowest to highest, wherever possible? Does anyone have experience with such instructions and could make suggestions?
Instructions and items will be translated to German, Slovak, Tamil and possibly to additional languages in order to do inter-cultural comparisons. So they should be very clear and straight forward.
In what final propositions are you interested? In other words, what are you looking for? Focusing on psychotechnical options (what to observe) without having identified the research goals seems inappropriate to me.
PS. Likert scales are ordinal, not quantitative scales. And the default is that two Likert-scaled items do not measure the same quantity. (see the link for a definition of measurement).
http://blogs.univ-tlse2.fr/vautier/1-a-definition-of-measurement/
My suggestion would be to create small scales of 4 to 6 items per topic, then present all items in random order. That way you could use a 5-point Likert scale, and order or context effects should be balanced out. Having multiple questions for a single topic would eliminate the problem of socially desirable responses, because (even if biased) little differences between ratings of different topics would be observed. However, a large sample would be needed to achieve a sufficient degree of power to detect these differences. A pilot study would also be needed to test the consistency of the scales.
I would not suggest ranking, because it limits your data analysis. Also, ranking presents even more problems: effects of the order of item presentation, context effects, effects of wording, etc. Balancing out these factors may prove quite difficult.
- Badges
- Science topic
- Similar topics
- Statistics
- Survey