Hi, I am developing a questionnaire (5 point Likert scale has been used) and I have conducted a pilot study. After checking, I see that a lot of responses are inconsistent.
For example, a respondent has indicated 'Strongly agree' on a positive item and on the 'reverse item (negative)' the respondent has indicated 'Disagree' instead of Strongly Disagree.
Are these responses valid for analyzing or should I delete this type of responses?
Does this means that I should use 3-point Likert scale instead of 5-point Likert scale?
Please give your suggestions.
Do not remove these responses. From your description, the items are showing consistency. The same individual is agreeing with a positively-phrased statement and disagreeing with a negatively-phrased statement.
You actually don't want too strong inter-item correlations. If you're seeing inter-item correlations around 0.9 - 1.00, this suggests that some items are redundant (i.e., asking the same thing). It is better to drop a redundant item than transform the scale.
Hello Moroo,
There are numerous studies which show that responses to ostensibly opposite polarity items are not exact complements of one another. In fact, sometimes they can be paradoxical (For example, try this pair of opposite polarity items: "Would you like to have a million dollars?" vs. "Wouldn't you like to have a million dollars?") That's just part of the panoply of things which influence a person's response to a verbal stimulus. If this is troublesome for your research, then perhaps you should avoid negatively worded stimuli (using these does increase the cognitive processing effort on the part of the respondent).
Trying to hide that lack of complementary relationships in responses by reducing the number of response options isn't going to gain you any precision. If the response options are intended genuinely to represent a continuum, then going from five to three options in general is a poor choice, as it would represent a loss of information.
Good luck with your work.
If you are going to combine the variables into a scale, it is standard procedure to have them all coded in the same "direction" (usually positive).
Later, if you wish to compare data from the pilot and sample you will see that it makes you difficult.
@Moroo I would agree with@David that you should not use negative polarity items as it often leads to inconsistent responses and the respondents sometimes might respond in accordance with the positive polarity items. I've personally experienced this problem in the early stage of my research journey suggest others not to have negative polarity items in their survey instrument.
Imran Anwar, Yes what you advise about not mixing positively and negatively worded items is now the widespread consensus on this issue.
Thank you so much everyone for the insights. The other issue in my pilot results is that there are acquiescence bias. In my opinion, I may solve this by taking a large sample in the main study and deleting those responses (acquiescence bias). I'm not confident this may work owing to low population (intended target population) and particularly since this is an online survey where in comparison to offline, response rates seems to be low.
Please give some more suggestions. I would be very grateful.
It is not appropriate to remove responses simply because they are inconsistent or may contain bias. My suggestion is to include a few distractor items to help weed out participants who are not reading the questions. If the issue is acquiescence, include some distractor items that anyone who actually read the question should choose strongly disagree. In the past, I've used items as dumb as "A dog is a type of reptile". Gathering self-report data online can be a headache. At the least, this will help you weed out participants who are just plowing through selecting 'strongly agree' on every item without reading them.
Regardless of what you do, explicitly state the method you're going to use concerning removal of participant data prior to data collection, justify the procedure, and report how many participants were removed.
P.S. Save the removed data somewhere! You may have occasion to use this data in future analyses. For instance, a reviewer might want to know how results change when including all participants - including removed ones.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence