I am analyzing responses to surveys where people have written very different text responses to the "please explain" text fields from their Likert scale x-choice in the same question? For example:
Q: How would you rate the responsiveness of home office to field office needs?
X choices=Excellent, fair, neutral, poor, terrible (respondent selected "fair")
*Same* respondent's text response in comment field = "Home office has almost never responded to any of my requests as field office director. My understanding from other ODs [office directors] is that they have an equally hard time getting home office to reply to requests, let alone get the request filled."
This is a strong and broad pattern across many questions and many respondents. I'm beginning to think I did something wrong to break the platform (SurveyMonkey) or just did a really poor job of constructing questions...if it is not me, what causes people to answer this way? Is this a real survey-based research thing or is it just me?
Hi Matt: I have not used Survey Monkey so I would not comment on the technical aspects. But there might some issues for such patterns.
1. The forced responses are not adequate to capture the respondents' sentiment.
2. Some respondents might have chosen the forced categories without reading, but written the explanation carefully or vice versa.
3. Is it possible that the respondents are choosing fixed responses of one question and writing the explanation in another question?
Good luck for figuring out the issue!
Cheers,
Nabin
Hi Matt,
Nabin explained some possibilities very well. In addition to that, if this is a considerable pattern in the responses, it sounds like there are broader issues at stake for your respondents, and the survey may have become a place to vent the problems they are experiencing, which may be nothing to do with the topic of the survey.
They will be expecting a response to their concerns if they have gone to the effort of telling you about them in the survey, so perhaps some follow up research is warranted to work out what the real issues are. To do this, you may need to do some qualitative research - face-to-face interviews could work well here.
Good luck!
Vicki
Interesting point Vicki! Sometimes people are interesting. When we did a survey in 2015, a guy sent us a 5-page letter describing why he thinks that the survey was flawed. In essence, he hated the topic related to the survey.
Nabin, that reminds me of something my supervisor said - sometimes you need to let people vent first, THEN you can ask them the questions you really want answers to! I've found that happens in the field during interviews, too. Some folks are so upset about certain issues that there's no real talking until they've got everything else off their chest. It can be tricky getting the interview back on track, though.
All the best,
Vicki
In the x choice questions, you are gathering mostly quantitative results, while the results of the open questions (explain) deal more with the qualitative aspects. I have performed research in areas such as job satisfaction in which qualitative responses seemed more truthful than the closed or forced - choice question.
However, last year we did a research on the youth of my country and responses of both types of questions were aligned. Thus, it seems to depend on the topic and characteristics of the population.
I agree with Vicki Martin that the best way to understand this apparent contradiction is to talk to the people who produced the data. So, I would follow her suggestion to do qualitative interviews to explore the puzzling results.
If the survey is done with personal interview, this gap is expected to be bridged. Usually , such a gap is more pronounced in Internet based surveys.
Hi
to prevent such discrepancies, you may recall in the label of the additional question what option was chosen in the Likert scale. Like: you answered "fair", please explain... Introducing interactivity, reminders of previous answers results in better data quality, according to my research works.
Good luck
Thank you all very much. Your feedback was great. My wife/biz partner said a lot of the same things -- we suspected and now we have confirmation! We hope to be able to move ahead with the interviews. A ton of findings from the survey analysis so far -- I just shared in my project update.
- Badges
- Science topic
- Similar topics
- Medicine
- Counseling Psychology