How many (what percentage) of 'unengaged' respondents would you allow in your respondents database and what criteria would you employ in order to make e decision to eliminate/keep?
With respect to Likert scales, there are times, especially in lengthy self-reported questionnaires, when respondents provide 'linear' answers, i.e., they give, constantly, the same rating to all questions, regardless if the questions are reversed or not.
Some call this type of respondents 'unengaged' respondents. This, is rather a 'friendly' term, since there can be also malicious intent in providing, but this is another discussion. However we call them, the first direct effect on the data is reduced variability.
There may be other effects as well (feel free to list them based on your experience or knowledge), which can affect the relations between the constructs and the final inferences.
Thus, how do you proceed in these situations and what criteria do you base your decision on (supporting arguments are welcome and references to written texts are especially appreciated)?
(Edit): I realised that the original question my induce some confusion. This is not a case that require substitution (although, in general terms, it may be). Please consider that all cases are complete (all respondents answered each item). The problem lies within the pattern of responses. Some respond in straight line, hence the name 'straightlined' answers, others in various patterns (zig zagging, for instance), hence the name "patterned". While for scales which include reversed items, some cases (for instance, 'straightlined") can be easily spotted, for scales without reverse items, this is harder to to. However, the question pertains both situations (scales with and without reversed items).
Another particularity of the question is that I am less interested in "how to identify" this cases (methods) and much more in "what to do"/"how to deal" with them, i.e., what criteria or rules of thumbs or best practices to consider.
Responses including references to academic papers or discussions are much appreciated!
Normally in Market Research Industry researchers allow 20-25% (Or 30% sometimes) unengaged respondents(usually they do mean substitutions). If missing is more than 30 & less than 50-60% they use appropriate replacement methods (for example : In case of linear Likert scale(5pointer) they recode their var which contains missing value as: top 2 box(4,5) with 1 & else = 0)
If variable contains 60-80% missing data points then use indicator function OR dummy variable for example in case for 5 pointer likert scale if a variable V1 has 70% missing values then we may use a recoding such as (1 to 5 = 1) and (rest all = 0) so the new recoded variable is binary & now using this variable in the analysis instead of that missing one.
In case if your variable has more than 80 % missing values, ignore that variable & do not include in Analysis.(However there are remedies in that case but that are not standard one).
Another check you should make before starting your Analysis is to check the standard deviation of the variables, try to ignore those variables which has approx zero sd i.e. variable is almost constant & hence it would have almost no use in statistical analysis (except in rare cases).
While designing questionnaire you need to be very much sincere, aware & sensitive in order to place questions & type of questions.
If you are doing any Analysis which includes formation of latent constructs etc...(SEM kind of stuffs) Ensure that you include enough variables which reflection of that construct. etc..
Let me know if you want any asst. in multivariate statistical modelling/projects.
hope it helps.
Rgds,
Abhay
It depends on the scenario. In industry in clinical trials we have to take all patient reported data into account even when patients fill it in with non-sense answers or answer it quickly as this is more common the reason for people not engaging with the instrument comments. I think in market research though it is good to filter people out as this in itself can be a sign that the survey instrument was not designed properly when people turn off or the topic is not of any interest. You can potentially force an answer out of people by getting an interviewer to administer the questions but you may be inducing a social desirability aspect where people feel they need to answer you and provide an opinion when they actually have not thought about it very much....In my experience it helps when you tell people what the data will be used for and you are enthusiastic about your topic. University schemes or employment surveys where people have to fill in information about work are generally not very informative - sometimes you just need to add a quick chat with people/ your target group to support and interpret your questionnaire research.
I use the standard deviation to identify unengaged responses. If the st. dev. is 0, then I exclude the respondent from the analysis. I am aware that this might pose a problem when there are few responses available.
- Badges
- Science topic