Hello
I'm working on the analysis of answers to a service user survey; since the service is highly tailored around the needs of the clients, most of the items have a Not Applicable choice in case a particular intervention was not relevant to the respondents.
I have a degree of otherwise genuine missing values (i.e. questions that some people chose not to answer) and I was going to use multiple imputation to deal with the missing data.
Since this is not strictly my field, I thought I'd sense-check what I was planning to do: as NA is a perfectly valid answer (and there'll be some work done with the clinical team to unpick what is behind the NA pattern that we see), my hunch would be to include NA as an imputable value in the MI. Even though I will still have a smaller population to run the contingency tables and logistic regression on afterwards.
Does this make sense or am I getting it very wrong?
Hello Laura,
My two cents' worth: yes, a N/A response can be a legitimate data point and therefore could be among the options for imputed values. But, if appropriate skip logic was built into a set of questions, that might obviate the need for doing so.
Good luck with your work.
Thanks for you thoughts, David Morse !
Unfortunately for these items we couldn't build in a skip logic - it would have made the flow unacceptable to the respondents and killed the response rate. But we're definitely going through a heavy revision of the survey approach, so next iteration should be easier in this respect
An approach that I have found valuable for multiple choice (radio button) responses is to use required response (no skipping) with "Other" as one of the choices, with a free text response (fill in the blank) displayed immediately adjacent or below "Other" with a caption label "If Other - please describe" with a character limit of 25 spaces. Free text fields like this should be left optional (not required). This method has worked quite well among multiple degree scientific faculty (a rather "picky" audience) who tend to perceive survey questionnaires as a waste of their valuable and precious time, if they ever see NA (not applicable) as one of the response options directed toward them! If the item is "NA" to me, then why are you wasting my time by even asking me a question that does NOT contain any way for me to give a meaningful response? Just putting that option "Other" (required) along with a fill in the blank (not required), cut the negative complaints on a annual survey to near zero from near 20% negative compared to previous instrument designs that had no such options. An increased response rate and an improved Cronbach's alpha was interpreted as "better engagement". Text analysis of the short 25 character free text comments using NVIVO and the IBM Modeler SPSS Text Analysis algorithms identified several semantic network "nodes" of similar concepts and themes among the shared responses, and these in turn were harvested and implemented as additional items among the multiple choices. In the final version of the survey instrument, "Other" and descriptions of "Other" converged to zero responses for that option and NO MISSING DATA! and very few complaints about the survey being "... a waste of my valuable time!".
Since then (about 20 years ago) I have used this method for when developing standardized psychometric instruments, and consistently seeing very similar results (less negative, more positive feedback from internal peer-reviewers) and more consistent and interactive engagement with the instrument itself among the target population of respondents.
Your mileage may vary, but this approach certainly works for me when I am trying to explore what "NA" really meant in the respondents mind, by making it possible for the respondent to tell me what it means, and literally in their own words, by requiring them to do do, and giving them exactly and immediately a limited way to do just that, through logic hard-wired into the instrument item design (combo question: required radio button option + limited open text), for each and every non-dichotomous (Yes/No) item in the instrument.
I hope this helps. This problem is complex, complicated, and universal to all psychometric instrument designs, as well as being a controversial topic in survey science.
Adding the information in this manner that is otherwise generally lost or discarded due to "skip logic" (in order to avoid required responses or to make it "flow better") adds about 30% more data (in my opinion) than you would otherwise have if you don't use this "sticky response" logic. "Sticky" meaning that you got more meaningful and useful psychometric information signal out of the respondent during the interaction and less meaningless "NA" noise and with a less negative end user experience! Win-win.
- Badges
- Science topic
- Similar topics
- Cognitive Science
- Thinking