I'm looking for advice from folks familiar with the research on overclaiming, social desirability or lying, or general measurement expertise, on ways of administering or scoring the overclaiming questionnaire (OCQ, e.g., Paulhus, Harms, Bruce, & Lysy, 2003).
In the OCQ, a participant is asked about their familiarity with items from various domains (e.g., historical facts, academic terms, famous writers or musicians). Some of the items are foils, i.e., made up. (Participants don't know that.) A participant who claims to be familiar with those theoretically overclaims/lies/answers in a socially desirable way. The questionnaire is typically scored using signal detection theory indices (accuracy, bias). But from what I've seen (it's a large body of literature!), the scale is always administered with a Likert scale (e.g., 1 = not at all familiar - 5 = very familiar). So to get at hits and false alarm rates, the scale has to be dichotomized after the fact. There doesn't seem to a strong argument for any particular cutoff value for which responses fall in the "familiar" and which in the "unfamiliar" category. Therefore, researchers often score it by using every possible cutoff and then averaging the results. For instance, they'll first score responses 1–4 as unfamiliar and 5 as familiar, then repeat the analysis using the 1–3 versus 4–5 cutoff, and so on, resulting in four scores per individual which are then averaged. That seems like a strange scoring system to me. It's not only cumbersome, I don't know if it's any more accurate than using one particular cutoff value. It seems pretty tolerant, for instance, to accept a value of 3 or 4 on a 5-point scale as NOT overclaiming one's familiarity with a completely made-up thing. Might a stricter cutoff not be more justified than the averaging technique?
My first question is: Does anyone know of a study systematically comparing different ways of scoring or different cutoff values?
Second: Why not use a dichotomous scale to begin with? I'm tempted to try that, but I'm wondering if it would suppress overclaiming, because people tend to lie in small ways.
I appreciate your advice!
SCALE. the problem with (1,2,3,4,5) is that at the lower end, 1 is a fake number that does not differentiate between a level 0 and level 1 of familiarity or expertise. As for 5, there is no way of telling whether level 5 is a true claim or false claim without a threshold point for reference.
THRESHOLD LEVEL. In order to verify the over-claim, there must be a threshold level or expected value for the expertise level beyond which a suspected over-claim may be imposed, i.e. separate the individuals from the rest of the data set and analyze separately. How to construct this threshold level? Find µ for the group of people with similar background and education. Set confidence interval for the threshold within a range of µ ± kσ where k is the number of standard units defining the boundaries of the confidence interval.
SIGNIFICANCE TEST. With defined threshold, the over claim and the normal claim may be separated. Compare the two groups to verify if the over claims are significant.
DICHOTOMOUS DATA. Do you have expertise in this area? (Yes | No). This approach will not cure the defect if there is an over-claim because respondents with such bias to over-state his/her expertise will score YES.
Hello Claire,
You're certainly chasing an interesting construct! I'm no expert in this arena, but, from a measurement point of view, the practice of looking at multiple thresholds and choosing the best (or averaging them) makes little sense unless one was willing to apply the "winning" threshold to new data to see if it continues to perform well. Otherwise, the obvious risk for overfitting, resulting in a biased and non-generalizable estimate of classification accuracy, is a very real concern.
As well, considering the 2003 version of the OCQ by Palhus et al., what floor does one set to avoid painting those who make clerical errors in response (perhaps due to haste to finish, or indifference to the task) as overclaiming persons? The ideal study would have an independent, external criterion against which to compare any derived OC indicator.
Keep us posted on your progress; I'd be interested in reading about any solutions you identify.
Good luck with your work.
Thanks for your answers!
Paul Louangrath, I appreciate your reply. To be honest, I can't quite follow. The OCQ is designed to measure the tendency for overclaiming, so I want there to be some level of overclaiming, I'm not treating that as a defect. I don't see the problem with a dichotomous yes|no scale, as long as participants still show the tendency to overclaim.
Since some of the items are foils, i.e., made up, I would certainly treat a level 5 as a false claim. The problem is with the lower scores, and finding a theoretically reasonable threshold. A comparison with the rest of the group might be a way to go, although theoretically they could all be overclaiming (especially since the foils are designed to sound compelling).
David Morse, thanks! I agree, an independent, external criterion would be ideal. I am planning to also use another task that measures actual performance and self-perception to get at that. As for the OCQ, I am leaning towards using a dichotomous scale or using a cut-off value that makes theoretical sense and pre-registering that value. (On a scale where 1=never heard to if before, 2=heard of it before, but can’t describe it exactly, 3=could describe it roughly, 4=could describe in relatively precisely, 5=could give an exact description, I think 3 is a reasonable cut-off.)
Look at how other researchers have delivered and scored the scale. Sometimes there are variations - other times a standard format. This varies as a function of the degree to which the original scale has been analysed.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics