Under the assumption of missing at random data, all participants that provide at least one response could be included in the analysis (no cut off) using multiple imputation or full information maximum likelihood estimation.

In the first step, it would be good to figure out (if possible) what the missing data mechanisms are (missing completely at random MCAR, missing at random MAR, and/or missing not at random). For example, you can examine whether there are any "auxiliary" variables in your data set that are correlated with the number of missing responses.

For MCAR and MAR data, multiple imputation or full information maximum likelihood estimation can be used to include all available data points.

I do not think there is a mathematically defined "breaking point" here. Test Manuals often provide guidance based on the authors' experience and judgment. Most folks would probably agree that if 10% of responses are missing, you're fine, but with 40-50% missing, the case should be deleted. If the scales you are using do not come with guidelines, you need to create your own and report them in any resulting papers. Personally, I would say that anything under 75-80% completion (i.e., an omission rate greater than 20-25% is unacceptable.

Important! Remember to pro-rate total scores when items are omitted! If a person only answers 8/10 items on a scale and they're on a 1-5 Likert-type scale, just ignoring the 2 missing items will have a pronounced effect on their scores. If, say, their average rating was a 3, failing to pro-rate would drop their total score from 30 to 24.

You say your 30-item scale is measuring "certain personality traits." How many? Because the number of items per scale is a big consideration here. If, say, you are measuring the "Big Five" or something similar, you have just 6 items per trait. At that level, anybody with more than 1 missing answer on any one scale should be dropped.

@Christian Geiser, I would assume the missing responses are MAR.

Would the interpretations of the analysis be somewhat compromised if we were to impute responses? The other factor at play is that these 30 items measure six traits, for which there are five items per scale.

I'm guessing imputation methods are more robust to larger item pools or scales that are composed of more items?

Thanks for the guidance!

Stephen Joy, thank you! This is very helpful.

I am measuring something close to the 6Cs, although there are only 5 scales in the instrument I'm analyzing, each of which only includes 5 items. So I suppose in this case it would be best to completely omit anyone with even one missing response to one of those 30 items.

You could make a case for retaining those with just 1 missing item on a scale and pro-rating. If you can spare those cases (i.e., there aren't very many), go ahead and drop them, but if they will make a real difference to your statistical power, then try keeping them. More than 1 item blank, though, is a definite no. Good luck!

Darko Giacomini The results are more likely to be "compromised" (biased) when you delete cases--regardless of how many items the cases are missing. This is because case-deletion methods such as listwise or pairwise deletion imply stricter assumptions about the missing data mechanism (i.e., they require MCAR data) than do modern methods such as multiple imputation and full information maximum likelihood (FIML) which only require the (weaker) MAR assumption. This means that you would potentially run into biased ("compromised") estimates when deleting cases even when the data are in fact MAR (but not MCAR). Case-deletion methods should no longer be used (see Enders, 2022, reference given below) because they are more likely to lead to bias and because they tend to result in less statistical power than MI and FIML.

Guidelines as to how many items need to be answered for cases to be retained are completely arbitrary and misleading. Instead, I would examine the mechanisms by which responses are missing and include as many auxiliary variables as possible when using multiple imputation or FIML. That way, you can retain as much information and power as possible, be more confident that you at least approximate the MAR condition, and minimize bias in the estimates. I recommend consulting a good book on applied missing data analysis such as

https://www.guilford.com/books/Applied-Missing-Data-Analysis/Craig-Enders/9781462549863

Christian - I wonder if we're thinking of different situations? Your scenario seems to be that of a missing data point for a participant (e.g., if they omitted to report their years of education, or whether they had a history of asthma), in which case yes, I wouldn't want to drop cases right and left unless the omitted item was the main focus of the study. What I'm thinking of is something operating within a single measure, where you have a number of items being combined to yield a score (which will serve as one data point). If enough items are omitted on, say, a personality test, the test can't be regarded as valid - there was either grave carelessness or a refusal to answer questions perceived as threatening in some way. I agree that opinions differ as to where to draw the line, here, but including invalid data doesn't seem like a good idea. The result wouldn't be to drop the entire case, necessarily - just the variable affected (e.g., a score on a trait such as psychoticism).

Stephen Joy

Christian Geiser

Thank you both for your extensive advice!

Just for further clarity, the missing points in my data set are occurring at the item level (which contributes to a scale).

So if items 1 through 5 contribute to a scale identified as Conscientiousness, participant 1 might have skipped question 2 (i.e., did not choose Strongly Agree - Strongly Disagree response option).

The instrument I'm working on is composed of 6 scales, each of which are measured by 5 items (so 30 questions in total).

I'm just worried that dropping the variable (i.e., item 4), would reduce the power/efficacy of my analyses.

Thank you again!

I would not drop anything (unless you have clear evidence that a participant provided invalid responses and/or showed a careless response style etc.).

Under the assumption of missing at random (MAR) or missing completely at random (MCAR) data, you can use multiple imputation (MI) to address missing data at the item level. MI allows you to retain as much power as possible because you make use of all available data.

Thanks for the clarification Christian Geiser

I'm in a bind as well because sample size is a bit of an issue as well, so I would want to keep as many cases as possible, so making use as much of the data as possible would be ideal. Will do some further research into MI and how it could be best utilized with my data.

Thank you again!

Missingness is always interesting, as well as being bothersome. When there are large numbers of missing responses, my immediate response is to look for missingness patterns. These can range from simple (stopped answering part-way through) to revealing and important.

As an example of the latter, we used a stigma questionnaire that had a whole clump of missing data. The missing data were concentrated in a small number of items which all related to work-related stigma. It was easy to cross-check : they were coming from people who were not in work. We omitted those items as not relevant to the whole study population.

Other missing items may just be vague or badly-worded so that even the data you have may be suspect – the people who did answer them were not really sure what they were doing. I recommend an item analysis using Mokken scaling to ensure that the items you are using form a definable scale, but even looking at item-rest correlations can help to spot these items. Actually – now I think of it – critically reading the items is the first thing you should do!

It's also possible that your missing responses are seemingly random rather than systematic. Perhaps, some individual respondents find some individual items threatening or personally inapplicable or some such. In that instance, I've always used the arbitrary rule of prorating scores, if people answer at least 80% of the items. For example, if somebody answered 24 of your 30 items, I'd multiply his/her obtained score by 30/24 = 1.25.