I performed a neuro-psychological battery consisting of 6 tests that measures different aspects of executive function. Each test has between 3 to 15 subscores. There is no consensus on whether one test or subscore is better than the other. There are a total of 64 subscores which are all non-normally distributed.
In this condition, is it appropriate to compare patients and controls using 64 univariate comparison using Mann whitney U and then perform Bonferroni correction or is there a more appropriate statistical model that I can use?
Hello Albert,
Sixty-four comparisons is a lot!
If you have the luxury of doing so (and in the absence of anyone else having so explored this), I would recommend factoring the subscores to determine what might be a more parsimonious set of factors/constructs that would account for the observed relationships among the subscores. Then, run statistical comparisons between groups on the factor scores. If factors are reasonably characterized as being inter-related, then a multivariate type analysis (see below for references to non-parametric multivariate tests) would make sense. If not, then a set of adjusted univariate tests could be conducted.
Good luck with your work!
References:
Hettmansperger, T. P. (1984). Statistical inferences based on ranks. New York, NY: Wiley. (See ch. 6, pp. 279-296.)
Sheu, C-F. & O'Curry, S. (1996). Implementation of nonparametric multivariate statistics with S. Behavior Research Methods, Instruments & Computers, 28(2), 315-318. https://link.springer.com/content/pdf/10.3758%2FBF03204789.pdf
(S code will frequently run without modification in R.)
64 comparisons will cause trouble. A Bonferroni correction will help, but it is also more conservative than necessary. However, in this case there is no way that I know of to correct that issue. If this was a single test with 64 treatments we would not use the LSD test with a Bonferroni correction because there are better alternatives.
One thing I look at is the probability of finding X number of significant outcomes in N tests given a fixed alpha. I then adjust my evaluation of how awesome my results are based on the probability that all I am seeing is an artifact of trying too many times.
Essentially you are trying to control the manuscript-wise error rate. So I would explore the 64 different comparisons, but I would also follow David's advice. Make sure that the multivariate method is appropriate for your data. Factor analysis is one option, but if you have defined groups discriminant analysis might also be useful. There may be other options depending on your data, goals, and hypotheses.
Your question is two-fold: How to deal with multiple tests with multiple items and how to analyze non-Normally distributed responses. Furthermore, I assume you have some sort of treatment. If this is so, you should definitely not use statistical tests (let alone 64), but parameter estimation.
1. For dealing with multiple test items, I suggest you estimate a multilevel linear model. For that purpose, you first have to make the sdata set long (one observation per row), like:
Part|Treatment|Test|Subscore|value
Now you can run the following model:
library(lme4)
M_1
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques