Since these questions don't seem to be 100% mutually exclusive and instead seem to be asking slightly different things, it may make sense that some respondents agreed with both statements. This may not necessarily be an indication of invalid responses, although it could also mean that some participants showed a careless response style, that is, agreeing with all items regardless of polarity. Maybe check whether these participants also agreed with most other items and whether that makes sense.

Latent class analysis can be used as a more formal statistical procedure to identify subgroups of participants that show a particular response style, for example, agreement or disagreement with all items. With LCA, you may be able to filter out all participants that showed a tendency to agree with most or all items. For an example application, see Eid and Rauber, 2000, European Journal of Psychological Assessment.

Eid, M., & Rauber, M. (2000). Detecting measurement invariance in organizational surveys. European Journal of Psychological Assessment, 16(1), 20–30. https://doi.org/10.1027/1015-5759.16.1.20

As Christian Geiser says, these are different questions so testing things like if the produce the exact same distributions or exact same associations with other variables is inappropriate. You likely want to examine if they are tapping similar response processes.

A common approach for this in this field (sometimes called "cognitive aspects of survey methodology" or CASM) is to use "think-aloud" protocols for people answering these questions (e.g., https://methods.sagepub.com/book/cognitive-interviewing/n4.xml).

Although these questions are not identical, I find it difficult to believe that someone could reasonably answer them at the two extremes (i.e., a score of 5 on each). If possible, I suggest that you examine outlier analyses for each of them separately (I say "if possible" because these are ordinal variables, so things like pearls correlations are not an option).

It seems like you want to test the internal consistency of the items of the questionnaire. In this case, I will suggest that you carry out reliability analysis. If you are using SPSS, there is an option that says, scale if item, check the box and run the analysis and see the value of Cronbach's Alpha. If it is low, then, delete the item that will make the value of alpha high.

There are some responses here which have a different reading of the question than I do, so I should expand on this. I am interpreting the question as: there are two questions, a and b, and these are being interpreted in such a way to yield A and B for the responses. A and B may be closely associated or not, from my interpretation assuming at least they are not identical (in their populations) it doesn't really matter. I am interpreting the question about whether a and b are being interpreted similarly. But this is different than Christian Geiser 's , David L Morgan 's , and Alhaji Modu Isa 's interpretations which appear to address the question as: "how to show if A and B are correlated". Siew Yuet Xin , perhaps you can tell me if I have just mis-understood your question. Since I am on the minority side of a 3-1 division I am assuming that I have!

Daniel Wright I do not understand your claim that two things "may be closely associated or not, from my interpretation assuming at least they are not identical (in their populations) it doesn't really matter." This sounds like saying that whether two things are strongly correlated or not doesn't matter, so long as the correlation is not 1.0.

So, you are correct in interpreting my position that if two variables are strongly correlated, then it is highly unlikely that people will answer them in exactly the opposite direction. Can you explain to me why it doesn't matter how strongly two variables are correlated?

Hello Siew Yuet Xin. I can well imagine different subjects interpreting those items differently. E.g., David L Morgan thinks it very unlikely that someone would respond 5 to both items. But if I believe that some small proportion of people can remain happy at all stages of PD, despite it being very difficult to do so, I might respond 5 to both items. So perhaps the items need to be revised. ;-)

PS- Not that it matters, but what is PD?

David L Morgan , what I am saying is that two things can be correlated and not be the same. I added the "at least" phrase simply because if by knowing one you know the other without error, then in some sense someone could say the refer to the same thing and some philosophers might argue this means that they are the same. The "in their populations" parenthetic clause is just that if you have a small sample and found r^2 = 1 in the sample it wouldn't be as good evidence of equality in the population than if n huge. My point was not about finding outliers, it was that previous responders were concentrating on what I have labelled A and B above, but I read the person asking about what I labelled a and b. From my interpretation (and given I am in the minority I am probably wrong), the questioner wants to know about how the respondents understood a and b and I suggested methods like think-aloud for that, which is used by some survey folks (and others ... e.g., we used it at ACT to see how kids would answer some of our questions).

Siew Yuet Xin

I like Daniel Wright 's think-aloud idea. After all, it could very well be that what you're seeing in your data isn't indicative of careless responding, but rather of "very careful" responding as pointed out by Bruce Weaver. That is, respondents who read the questions carefully may have noticed the difference in wording and shown a highly differentiated response. These two questions do not measure/ask the exact same thing, and so you cannot expect respondents to show perfect consistency in their responses across the items. (Perfect consistency here may even speak against the validity of the items since they measure slightly different things.)

One way to find out is to look into how these same people responded to other questions (careful or careless?). Statistically, this can be done using latent class analysis / mixture modeling (which can identify homogeneous subgroups in which participants show a specific response style), provided the sample size is large enough for such an analysis (say > 300 or so). An even better way may be to ask people directly as suggested by Daniel Wright .

Hi Christian Geiser , Daniel Wright , thank you for the suggestions , I will look more into LCA. But I would consider changing these questions as it was causing confusion