Hello Melissa,

If your immediate aim is that of trying to identify a plausible factor structure for your new measure, then the brief answer is to rely only on the scores from that measure (and not the demographic attributes).

However, there are a couple of points you may wish to consider.

First, as the individual responses on the target measure are ordinal strength, you'll need to consider how best to proceed; treating them as interval strength (which is what both the computation for distance measures and the subsequent factoring would do by computing covariances/correlations) is likely not the best choice. Some options here would be: (a) using polychoric correlations; (b) using IRT to explore both dimensionality and unusual (non-model-fitting) cases; (c) using some other scaling method, such as Guttman, S-P (subject-problem), or Mokken; or (d) using factoring models that allow for ordinal scores.

Second, a lot of folks may (rightly) question why you think that outlying cases should be jettisoned from your data set. Cleaning data of impossible values or miscoded scores is an obvious step. Deciding that your sample can not have atypical cases, albeit with otherwise legitimate scores is not so obvious a need or requirement.

Good luck with your work.

When conducting Mahalanobis distance analyses to search for multivariate outliers, it is generally recommended to include all variables in the analysis, including both the target variables and the independent variables (IVs). This means including the variables that represent the item pool for your psychological safety (PS) assessment as well as the demographic variables, existing PS assessment, interdependence measure, etc.

Including all variables in the analysis allows for a comprehensive examination of multivariate outliers, considering the relationships among all the variables in your dataset. This approach ensures that potential outliers are identified based on their multivariate distance from the overall data distribution, taking into account the entire set of variables.

Excluding certain variables from the analysis may lead to biased results, as outliers in one variable may not necessarily be outliers when considering the multivariate space. By including all variables, you are better able to capture the overall structure and patterns of outliers within your dataset.

In your case, it would be appropriate to include all the variables, including the PS assessment items, demographic variables, existing PS assessment, interdependence measure, etc., in your Mahalanobis distance analyses. This will provide a comprehensive assessment of multivariate outliers in relation to the entire set of variables you have measured.

Therefore, it is recommended to use the syntax you provided, including all the variables, to conduct the Mahalanobis distance analyses and identify multivariate outliers.

Thank you so much, both of your answers were extremely helpful! I ultimately made the decision to run the analyses 3 times and then compare results - (1) full data set, (2) with n = 10 multivariate outliers removed, and (3) with n = 28 multivariate outliers removed. It was time-consuming, but luckily there were very few differences in results, so I didn't have to make any hard decisions.

I will have to look into the analyses David Morse suggested. In selecting my rating scale labels, I used research performed by Beckstead (2014). This study attempted to map the strength of adjectives that are typically attached to Likert scales - thus allowing researchers to select rating scale labels that are "spaced an equal distance apart," and thus to proceed with analyses under a crude assumption of interval scaling. But given this spacing assumption is based on single study, its hardly satisfactory.

Thank you again both of you!