For example, in my questionnaire the respondent indicated that their dog spent 2 hours a week interacting with other dogs, and then 2 questions later they said that their dog interacted with 0 dogs per week. This is during the same time period. To me this indicates that the person completing this questionnaire must have answered at least one of the two questions incorrectly in which case I would be inclined to exclude that respondent from the entire study. I have tried to find examples of this in the literature but cannot. Any advice greatly appreciated, I have researched internal consistency but it doesn't seem to apply to this.
Dear Dennis, I think that it is usual to formulate questions in a different way to detect whether the respondent is honest or not. Somehow the amount of contradictory answers is evaluated also. But I don't know how exactly. There must be studies about this and surely a sort of "correction" terms in the stadistic. But, I think you cannot discard the respondent for his contradictions. Maybe an expert in sociology and questionnaires can give you a more accurate explanation of how to evaluate the contradictory answers. Best regards, Alexandra
If you cannot redesign your questionnaire you should certainly not include contradictory data in your analysis as it is invalid.
I agree with Penny that you should systematically check all the respondents and devise a standard for those you exclude. In this case, it seems that you have asked two versions of the same question, so you can correlate them (in conjunction with a scatter plot) to search for "outliers" who gave inconsistent response.
Thank you all so much for your replies it really helped. Martin that paper is excellent and is just what I was searching for. It describes what I have done as "iterative questioning" which helps establish credibility of responses. I will certainly check the entire dataset Jane and exclude all spurious cases which are self contradictory. The idea of doing a scatter plot is also excellent thank you David.
i can see that there is problem in the question you framed. if more than 20 per cent have answered like this better, remove that content.
Great discussion here. Also, it seems that inconsistency would be reflected in the alpha reliability test for internal consistency. Also, a good source for scale development is:
Abell, N., Springer, D.W., Kamata, A. (2009). Developing and validating rapid assessment instruments, Pocket Guides to Social Work Research Methods series, Oxford University Press.
the authors also have a series of peer reviewed publications on the topic that have proved very useful in the past.
When the same respondent gives contradictory response, that means this respondent should be excluded from the analysis. The answer from this respondent will afftecs the validity and reliability of the study.
I think it is an ethical issue. For example, if my Q3 is "have youheard of the term 'global warming'"? with Yes/No options provided and an instruction to "skip to Q5 if your answer is NO". If a respondent answers NO for Q3 and goes on to answer Q4 on the causes of global warming (which he is not supposed to answer), it will be difficult for me to decide to discard Q4 for this respondent and analyse the rest of the questions for him. The dilemma is that I cannot not be sure whether the respondent had actually heard the term "global warming" but had mistakenly chosen NO in Q3, in which case I should change his response to Q3 to YES and analyse Q4 for him. This will affect the integrity of the research and for ethical reasons I will discard the questionnaire for that respondent
Hi Dennis,
on the level of quantitative statistical methods there are 3 possibilities:
- deleting that item
- using the mean of both answers
- deciding that either answer 1 or 2 are incorrect and using only the correct one
Statistically, option 1 is safest. Depending on your population size that probably won't change the results and you've got "clean" data.
However, if your population size is small or contradictions happen frequently, just go with the answer of Martin.
Best, Oliver
This means that your respondents were not taking your questionnaire seriously. If i were you, i would consider reviewing their whole answers. Also, if you did not finish collecting your sample yet, then i recommend using a two or three attention filter (e.g, Please choose (1) strongly disagree for this question etc..).
Well yes, I think that is fine particularly in qualitative research but also in survey work. There are contradictions - there is what you would call your question order and that could precipitate quite a different answer in one part of the questionnaire than the exact same question but somewhere later on. We always try to 'force' consistency. I don't think that is the right way to go. Would love to hear your response. Jack
I agree with Jack, we should not be trying to force consistency but should not these cases especially if they exceed a certain level (see below). Contradictory responses of this type are not so unusual in survey responses and I do not think it necessarily means the respondent is not taking the questions seriously. On the contrary they could be taking it very seriously and (just one example) reinterpreting 'interact' as they proceed through the questions. Something depends on how frequently this type of response arises. If it is arising in say 1% of cases or above then I would question the piloting of the survey instrument itself.
- Badges
- Science topic
- Similar topics
- Publishing