Hello Ale,

You can take such items and treat them normatively; doing so requires specifying one (or several) of the options as the 'target' endorsement, and giving credit for those selections, but no (or less) credit for other selections. From your example, if the operative construct was that of teamwork/cooperation, then choices such as (b) in the sample item would be given credit whereas options like (a) would not. Then, summated scores over the set of items could be determined and cases sorted based on their standing relative to the other respondents.

The downside of this practice is that summation works best when the set of items is unidimensional. If multiple constructs are involved, the scores become more "noisy" and less useful. Also, item response models may be used, with the same caveats. The other downside is that perhaps the ipsative interpretation may be more meaningful in practice than the normative interpretation, though that would depend on the intended use(s) of the scores.

Good luck with your work!

Hi Ale,

there are recent developments in IRT that allows extracting normative scores from forced-choice questionnaires. The Thurstonian IRT (TIRT) model by Brown and Maydeu-Olivares and the MUPP model by Stark and colleagues are good examples.

Article Item Response Modeling of Forced-Choice Questionnaires

Article A Dominance Variant Under the Multi-Unidimensional Pairwise-...

From my own experience, the TIRT model works best in practice (i.e., in terms of reliability and validity).

Of course you can derive norms. I don't think there is any question on that count. The issue that can arise relates to the use of standard correlational analyses and anything built on a correlational foundation (e.g., multiple regression, factor analysis).

If you are speaking of a single-scale measure in which each item consists of two contrasting statements, one indicating one end of a continuum, the other the other, then there is no statistical issue. For instance, an introversion-extraversion scale might be constructed in this way, with items like "I enjoy playing soccer" vs. "I enjoy cross-country running." You can still run into conceptual difficulties with a factor analysis, because it's often the case that one end of the continuum is better defined than the other. A classic example would be the Rotter locus of control scale; a perfectly good measure, but subsequent analysis suggested that the "external" end comprised several distinct concepts ("powerful others," "chance," and offhand I forget the third). This led to the development of Levenson's scale, now more widely used than Rotter's.

Where the statistical difficulties arise is when you are measuring multiple constructs, contrasting each with each of the others. The classic example here is the Edwards Personal Preference Schedule. This was designed to measure motives. As with values, the strength of a motive can be understood only relative to other motives. For example, everyone would like to go on a date with an attractive person, and everyone would like to be recognized for an achievement. But which would you rather have, if you had to choose? So the Edwards was built of statements exemplifying a range of motives, with items contrasting every possible pairing. (The Minnesota Importance Questionnaire does something similar work work-related values.) It's basically an excellent idea. But the trouble is that the ipsative scores so derived aren't truly independent of one another. If, say, you have 10 scales, then once the scores for 9 of them are established, the 10th is known. This renders any correlational findings a bit suspect (though honestly, it probably doesn't invalidate them except in the eyes of a statistical purist) and can really ruin multiple regression or factor analytic analyses (up to the level of generating error messages from the program used for the analysis).

One work-around I've seen is to add an extra "dummy" scale, a set of options that do not get scored. I'm not sure this would pass the smell test for a serious statistician, but it seems to me I've seen it used by people who should know.

Well, this got long, didn't it? One last thought: in my opinion, the forced-choice method has real merit for certain purposes. It "fell from grace" largely due to the emphasis on factor analysis and the notorious statistical problems posed by the Edwards scales. But it deserves a second look.

Dirk Pelt:

Thank you for the answer.

I've been looking at the articles. It's a lot of theory:)

Do you know of any studies in which TIRT is applied in a practical way? because I have seen several researches in which they make simulations, but I have not found any that use it in a practical way, for example in the creation of a test and later generating normative scores.

David Morse:

Thank you for your help.

The context in which I would like to use it is in recruitment and hiring, in these contexts it is more meaningful to make normative interpretations.

I think some traits can be measured correctly in a one-dimensional way (as in the example you gave me) since either of the two answers is acceptable. However, the problem comes when there is no "acceptable" opponent for the trait. For example, to find an opponent of responsibility who is socially desirable, I think it is impossible.

This is the basis of my problem, measuring traits in hiring contexts by properly managing the problem of social desirability.

Now Sebastiaan's stealing my answer! Yes, something along lines of risk-taking (or "derring-do") would be a good "counterpoint" to a Responsibility (or Conscientiousness) item. Like if you contrasted (a) "It is important to follow accepted procedures and document your work" with (b) "A good project manager knows when to throw the book away and improvise."

The interesting thing about Stephen Joy's last answer above is that I would find it difficult to reject either response. And what he said in his long answer is rather similar to replacing 'project manager' with 'psychometrician' in (b) - while your typical statistician would reject that and stick firmly to (a). Unfortunately, in many cases the solution you chose will be dictated by the editors of the journal you want to publish in. Given that most of our psychometric solutions yield only an approximation to the truth, and are rapidly forgotten (cf Edwards' scales and numerous others), I would recommend trying to be 'a good psychometrican.'

Hi Don - Making the choices nearly equivalent in desirability is one of the real advantages of the forced-choice approach. The respondent has to ask, "which of these is closer to my views?" or in some cases "which of these is less antithetical?"