Unfortunately, we just realized after data collection, when we wanted to start analyzing our data, that some items were missing in the online survey. The survey consisted of three parts (three already existing questionnaires): the AUDIT (Alcohol Use Disorders Identification Test), the UCLA-3 Loneliness Scale, and the FFMQ (Five Facet Mindfulness Questionnaire). For the last one the problem occurred: it should have consisted of 39 items, but our data set consists only of 21 items; it seems as if we simply forgot to include 18 items and nobody realized. This is a big problem, right? Validity and reliability of the FFMQ are not the same anymore, obviously.
Could anyone help us? What should we do now? Explain in the methods section of our research paper that we chose some items and not included all 39 on purpose? Is there even an explanation to do something like this?
Thank you in advance!
Two ways: 1. Redo at least parts of the survey or 2. Data imputation. However, it is likely that the responses are NOT missing at random so keep that in mind if you choose imputation. Best, D. Booth
David Eugene Booth Thanks for your prompt response! But how to redo parts of the survey, if it was an online questionnaire administered via the University's participant pool? 162 (anonymous) participants took part.
What do you think about the possibility of just doing a reliability analysis ourselves, to check whether the 21 items of the FFMQ are already reliable enough to be reported?
That sounds like a reasonable approach. I've seen good advice on things like that from @David Morse. You might check with him.
I think your first step is to be completely honest in your reporting. Mistakes happen in research and you should acknowledge this in the Limitations section.
It's obviously not the same scale so while you can analyze your data your interpretation will be more tentative. Can you compare the data on the items you did collect to previous publications to see if at least your means and SDs are 'close' to the others? When you do your analyses with your incomplete data, were your findings at least in the direction you hypothesized?
As David said your missing items are not likely missing completely at random. If you left off (say) an entire page, and all those questions formed part of a sub scale, then you might actually have complete data on some sub scales and no data on others. In this case the (complete) sub scale data are analyzable as is but it would be very difficult to impute values of a completely missing sub scale from complete data on other subscales.
Finally, you are probably aware that an active area of research is to create shorter versions of existing scales. (E.g. SF36 - SF12, etc.). So another thing you could try (unless you are missing one or more entire sub scales) although it wouldn't be easy would be to contact an author who would share data on the complete scale in a large sample so you could do some comparative analyses: how close is your "FFMQ-21" to the FFMQ-39?
None of these are ideal. The major question is whether your missing items are entire sub scales or if the sub scale items are distributed evenly throughout the scale. I don't know anything about the FFMQ.
Good luck.
Mark R Speechley Thank you so much for your detailed answer, that is very helpful!
Luckily, not a whole sub scale is missing. In our survey, every sub scale (5 in total) is represented with 3 or 4 items; normally it would have been 7 or 8 items. I tried running an reliability analysis and it seems as if it's still good, but the analysis is not complete yet. We'll compare our data to previous publications as you recommended and see whether we can also do more than that.
This depends on the stage of your study. If you haven't concluded your study, you may still have the opportunity to redo and make changes. Otherwise, you just have to be honest in your report to acknowledge it as a limitation
There are several approaches for treatment of missing data but it all depends on how much of the data is missing and the pattern or mechanism of the missing data. Since your missing data on the specific scale is more than 10% which some authors suggest could be ignored, I recommend you consider using some principled approaches. You can read more from : Article Principled missing data methods for researchers . All the best.
Share
I have heard of analyses where respondents were treated as "missing at random," but I have never encountered an argument that items in a scale could fit this pattern.
If all the items in the original measure were intended to measure a single concept, then doing a new reliability and validity analysis with your truncated version makes good sense. But if there were supposed to be sub-scales within the total set of variables, you are likely to have problems reproducing them.
Clearly you need to be honest. I suppose there is an argument that you don't need to admit a mistake, and just say (don't say intentionally) that your survey incorporated 21 items from a 39-item scale, but someone is going to ask you how and why you chose the 21 items. It's better to say something like: Through a production error, 18 items were omitted from the original 39-item scale.
Then proceed to explain what you did in response: what checks you made to determine if all sub-scales were present, along with an analysis of the scale's and sub-scale's reliability. I wouldn't recommend imputation unless you have additional data that might usefully be brought in. Otherwise you are not really adding any new information to the scales.(Plus, the new data shouldn't be critical data that you plan to use in your analysis, because then you might build biases into your analysis.)
Of course, you could also redo the study from scratch, but I hate to throw away data. You might as well see what you have.
I'm not sure how you would impute the values for an entire variable, let alone 18 of them.
If that section missing is important, then, simulation might help, but if the section address a hypothesis, then remove that hypothesis from the work if it does not affect the research if you do do.
Otherwise post the complete set to the university online platform, and set a time limit for the response. This might either increase the sample size or reduce it. Hope one of the suggestions helped
- Badges
- Science topic
- Similar topics
- Statistics
- Survey