What you are referring to are the so-called ipsative tests. In general, as far as I know, there are not many ways of studying their psychometric properties. The reliability is usually estimated using the stability method (test-retest). The following article can give you more information: Article The Reliability and Validity of Normative and Ipsative Appro...

Best regards

What you are referring to are the so-called ipsative tests. In general, as far as I know, there are not many ways of studying their psychometric properties. The reliability is usually estimated using the stability method (test-retest). The following article can give you more information: Article The Reliability and Validity of Normative and Ipsative Appro...

Best regards

You could check:

An Alternative Approach to Analyze Ipsative Data. Revisiting Experiential Learning Theory. doi: 10.3389/fpsyg.2015.01742

How IRT can solve problems of ipsative data in forced-choice questionnaires. doi: 10.1037/a0030641

I think the Calero response is a good one. However, there is also a way to look at the internal reliability of an instrument such as this if it is constructed appropriately. I am assuming that the purpose of this instrument is to construct a hierarchy of an individual's characteristics. Internally, there should be at least two items that cross-check each characteristic. A simple way of doing this is to include the same item twice in the instrument randomizing the order of responses each time. Assuming you had a 100-item instrument, you could correlate the 50 pairs.

One major caution! Do not try to check validity or reliability of an ipsative instrument using factor analysis. I have attached a paper that explains why you should should not use factor analysis with ipsative data. The paper is also available on ResearchGate.

In the construction of standardized psychological instruments - inventory of personality - frequently used multiple-response items. The multiple-response format consists of a question or sentence called the premise, followed by a number of responses - typically four or five - of which ONLY ONE is correct. Incorrect responses are called "distractors" and must be built in such a way as to appear fair to a person who does not know the answer. When writing an item with multiple responses, it is important to clearly differentiate the correct distractor response, otherwise it can record errors in the measurement process. Considering the requirements presented, psychometric instruments do not allow the construction and use of items with several different correct answers. These items can only be built for knowledge tests, but without psychometric value (fidelity, validity, etc.).

thank you all for your help. According to my research, it is almost impossible to make comparisons between subjects through ipsative tests. Therefore, it would be best to use likert scale items (in this case the problem is related to social desirability in the answers)

Ale Jo:

You might find your solutions in the Partial Credit Rasch model. This model assumes that each possible response has its own structure and may be valued (scaled) differently from any other response in the particular item. Other graded response models, such as a Likert rating scale, on the other hand, assumes that each response point is equidistant from corresponding points in other items. This certainly is not likely in the case of the data you present, nor is it likely to have equally appearing intervals.

If you are working strictly in classical test theory, you might conceptualize each item response as a separate binary item. Given your constraint of "select one" any sense of local independence of items is severely violated and it may be difficult to determine what you have. The dynamic structure of your data may make a satisfactory analysis intractable.

If there is some underlying structure to the data, i.e., each "distractor" represents some dimension of the data, then you might try to hold it constant and examine data only when it is not selected. Then you have reduced the data to two possibilities rather than three. You can then score the dominant category relative to the subordinate category.

If this is test data, what is it testing? I am wondering what you want to measure (how much) or whether this is about a classification problem (what kind.) In any event, it seems like a hodge-podge of numbers that may not have a theoretical framework of how they are expected to inform you. What is your theory of the problem? Perhaps it would be useful to keep in ind that numbers without structure are pretty ambiguous. Consider that 2 people and 24 bottles of ber do not produce the same results as 24 people and 2 bottles of beer.

I prefer to use a program called "Winsteps" for such analysis in estimating parameters of a hypothesized structure, but there are other programs that may be equally appropriate. I think there is a program written in "R" that may work for you. In any event, use your search engine to explore "Partial Credit Rasch Model."

Best wishes,

David Drehmer

You could score a particular combination of answers to questions as best, another combination of answers as second-best, and so on, until you have assigned a ranking to each permutation of answers.