Your issue involves missing data by design in a multilevel logistic regression setting, where participants provided data on only a subset of the possible selection procedures based on their usage (use vs. don't use). The missing data pattern you described is systematically missing (i.e., not missing at random), as the design of the study itself dictates which data points are missing. This kind of missing data needs careful consideration when performing statistical analysis.

-Handling Missing Data in R:

R, like HLM, is quite capable of handling missing data, particularly when the missingness is part of the design and not due to random participant dropout or missing responses. Given your specific scenario where you have structured non-response (every participant rates exactly four out of five procedures), you can handle this in the following ways:

1. Use available data: Since your data are missing by design, you can proceed with the analysis on the available cases for each selection procedure. The multilevel logistic regression model can be fit to these data, acknowledging that each individual provides responses to only a subset of the items.

2. Model specification: In R, you can use packages such as `lme4` or `glmmTMB` to run multilevel logistic regression. These packages can handle missing data as long as the missing data mechanism does not violate the assumptions of the analysis model. Here, since the missingness is structured and non-random, these models can accommodate that by fitting the model only on the observed data points.

3. Coding the data: Make sure your data are structured such that the model correctly interprets the absence of data for a given procedure for a given participant as a planned missingness rather than a randomly missing point. This often involves ensuring your data frame is correctly formatted and only includes rows for which data exist.

-Example in R:

Here is a very basic outline of how you might set up a multilevel logistic regression in R using the `lme4` package:

.................................

R

# Install if not already installed

# install.packages("lme4")

library(lme4)

# Assuming your dataset is called `data` and has columns for participant ID (`participant`),

# procedure used (`procedure`), binary use/don't use (`use`), and outcome (`rating`).

# Ensure that data only includes the cases where a rating was given.

# Fit a logistic regression model

model

Thanks so much Saadi Achour

I appreciate the recommendation to do the sensitivity analysis especially as it's something I wouldn't have thought of.

I'm running into an issue now with the CFA - would you have any recommendations for that in terms of the missing data?

Here is the current code I am running:

>m1a < - 'f =~ SS_1 + SS_2 + GMA_1 + GMA_2 + PT_1 + PT_2 + AC_1 + AC_2 + SM_1 + SM_2'

>onefac10items_a < - cfa(m1a, data=rdata,std.lv=TRUE,estimator="MLR")

>summary(onefac10items_a, fit.measures=TRUE, standardized=TRUE)

IYH dear Stella Buley

The good news is that yes, there are specific ways to handle missing data in your case. Since you have a binary independent variable (IV) and participants were asked to rate multiple selection procedures, you can use so-called multiple imputation techniques to handle the missing data, which is a robust and flexible method for handling missing data in statistical analyses.

Multiple imputation involves creating multiple copies of your dataset and replacing missing values with plausible values based on other information available in the dataset.

Here are the steps to handle missing data using multiple imputation:

1. Determine the pattern of missing data: Identify the missing data pattern and determine whether the data are missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). Standard is to assume MAR.

2. Choose an appropriate imputation method: If the data are MAR or MCAR, you can use multiple imputation methods such as chained equations or predictive mean matching. R has the mice package. https://cran.r-project.org/web/packages/mice/index.html

These methods use regression models to predict missing values based on other variables in the dataset.

3. Impute missing values: Create multiple copies of your dataset and replace missing values with plausible values based on the imputation method you chose.

4. Analyze each imputed dataset: Run your multilevel logistic regression analysis on each imputed dataset separately.

5. Combine results: Combine the results from each imputed dataset using Rubin's rules to obtain a single estimate and standard error.

Rubin's procedure using R is explained here https://www.lib.virginia.edu/data/articles/getting-started-with-multiple-imputation-in-r

I'm not sure you need to impute. Multilevel models can cope with missing outcome data without any imputation (basically this is just imbalance and the model is treating the missing responses as if MNAR). There is an issue here letting participants choose which option not to respond to - as that could bias things. If the choice were balanced or random then it would be better than letting them choose.

IYH Dear Thom Baguley thank you very much. I ran your SME comment against my bespoke expert stats AI bot and he concurred w your assessment for the query of the poster https://hf.co/chat/r/bSIcKcf

tl/dr "In summary, while imputation is a common technique for handling missing data, it may not be the best approach in your specific case due to the intentional nature of the missing responses. Multilevel models can handle missing data and provide valid estimates, but it is crucial to acknowledge and address the potential bias introduced by the non-random missingness. Sensitivity analyses and a careful examination of the missing data patterns will help ensure the robustness and validity of your findings."

Prompt:

Q1: Is there a particular way to handle this kind of missing data? Imputation comes to mind

Q2: My SME colleague says he is not sure one even needs to impute. "Multilevel models can cope with missing outcome data without any imputation (basically this is just imbalance and the model is treating the missing responses as if MNAR). There is an issue here letting participants choose which option not to respond to - as that could bias things. If the choice were balanced or random then it would be better than letting them choose."

What's your opinion w rationale please ?"

Output:

Hello there! I am Yissachar Yehudi, an expert statistician with a diverse skill set. I can assist you with your queries regarding missing data and statistical analysis.

Q1: Handling Missing Data:

In your case, where participants were asked to rate selection procedures, resulting in missing data for one of the procedures, imputation is indeed a commonly used technique to handle missing values. Imputation involves substituting the missing data with estimated values based on the available data. There are various imputation methods available, such as mean imputation, regression imputation, or more advanced techniques like multiple imputation using chained equations (MICE). The choice of imputation method depends on the nature of your data and the assumptions you can make about the missing data mechanism.

Before opting for imputation, it is essential to consider the pattern of missing data. If the data are missing completely at random (MCAR) or missing at random (MAR), imputation can be effective. However, if the data are missing not at random (MNAR), as your colleague suggested, imputation may not be the best approach. In your case, since participants intentionally chose not to respond to one of the procedures, the data are likely MNAR, and imputation may introduce bias.

Q2: Multilevel Models and Missing Data:

Your colleague is correct in stating that multilevel models can handle missing outcome data without imputation. Multilevel models, or mixed-effects models, are designed to handle nested or hierarchical data structures and can accommodate missing data under certain assumptions.

In your case, the missing data are due to participants' choice of not responding to one of the procedures. This introduces a form of selection bias, as it is not random but rather dependent on the participants' preferences or characteristics. Treating the missing responses as MNAR in a multilevel model assumes that the missingness is unrelated to the outcome, which may not be valid in your case.

Opinion and Rationale:

Given the nature of your data and the research design, I agree with your colleague that imputation may not be necessary or even advisable. The fact that participants chose which option to not respond to introduces a potential source of bias that imputation may not adequately address.

However, this does not mean that you cannot analyze the data effectively. Multilevel models are robust and can handle missing data when appropriately specified. By treating the missing responses as MNAR, you acknowledge that the missingness may be related to unobserved characteristics. The model can still provide valid estimates for the fixed and random effects, assuming the data meet the assumptions of the model (e.g., normal distribution of random effects, homogeneity of variance).

To strengthen your analysis, I recommend the following:

- Sensitivity Analysis: Conduct sensitivity analyses to assess the impact of the missing data on your results. Impute the missing values using different methods (e.g., mean imputation, multiple imputation) and compare the results with those obtained from the multilevel model without imputation. This will help you understand the robustness of your findings to different assumptions about the missing data.

- Explore Patterns in Missingness: Examine the patterns of missing data and their relationship with observed variables. If there are systematic differences between participants who provided responses for all procedures and those who did not, it may indicate that the missingness is not random and could bias your results.

- Consider Alternative Models: Depending on the research question and the structure of your data, alternative modeling approaches, such as selection models or pattern-mixture models, may be more suitable for handling the missing data and making appropriate inferences.