Hi James,

Maybe I missing something but I have some concerns with the "run both procedures approaches". I think it is always worth considering multiple approaches and exploring possibilities, but I think this should be done with care. I should add, with the focus on work like Simmons, Nelson & Simonsohn (2011, Psych Sci), I have become more sensitive to the use of this approach.

I'll give two of these concerns.

The first is easily dealt with. If you are running two hypothesis tests for two similar comparisons, then you have to correct for this. This makes this approach (depending on how you adjust for the multiple tests) less powerful than either in many circumstances. But unlike a lot of literature I don't think the power of the test should be the most important thing to base one's decision on, but on whether the procedure is appropriate for the research question and design (and then the assumptions bit).

The second concern is that the two procedures are testing different things. A good description of this is in David Hand's great paper "Deconstructing Statistical Questions". This is his example 4 (http://statlab.bio5.org/sites/default/files/fall2014/hand-deconstructin.pdf). His point is that the two approaches are addressing DIFFERENT research questions. If the researcher doesn't know which they are asking, Hand's argument is that they should figure that out before doing the analyses (or before designing the study). At least wrt causal inference, Wainer, Pearl, etc., discuss when the ANCOVA approach is more likely to be valid.

I think it is a slippery slope running multiple hypothesis tests for different research questions and then selecting "the most powerful" (which I assume means lowest adjusted p value here), and having that determine what your research question is. I would think the research question should be known beforehand, and not determined by p values (Gelman and Loken make similar arguments in their garden of forked paths paper).

Again, I might be missing something, so correct me if I have read your advice incorrectly.

Your example is related to Lord's paradox, though your example is more complex than his. Most agree that for causal inference (here, that your invention makes a difference) that the MANCOVA should be used. Wainer (1991, Psych Bull) describes in relation to the potential outcomes model of Rubin and Neyman, Pearl (2014, on his webpage, google Pearl Kumbaya) describes in relation to graphs, and the attached paper uses simulation methods.

Article Comparing groups in a before-after design: When t test and A...

Daniel's argument seems reasonable.

The crux of your question is which is more powerful in a repeated measures situation, multivariate or repeated measures analysis.  I have not looked at this in detail for about ten years, but at that time, I found that about 1/3 of the time, there was no difference.  In about 1/3 of the time a repeated measures analysis was more powerful and in about 1/3 of the time the multivariate analysis was more powerful.  From that experience, I would tell my students to run both a repeated measures analysis and a multivariate analysis if the basic assumptions are met and select the most powerful.  The advice given by Dr. Wright should also be considered in this decision.

Hi James,

Maybe I missing something but I have some concerns with the "run both procedures approaches". I think it is always worth considering multiple approaches and exploring possibilities, but I think this should be done with care. I should add, with the focus on work like Simmons, Nelson & Simonsohn (2011, Psych Sci), I have become more sensitive to the use of this approach.

I'll give two of these concerns.

The first is easily dealt with. If you are running two hypothesis tests for two similar comparisons, then you have to correct for this. This makes this approach (depending on how you adjust for the multiple tests) less powerful than either in many circumstances. But unlike a lot of literature I don't think the power of the test should be the most important thing to base one's decision on, but on whether the procedure is appropriate for the research question and design (and then the assumptions bit).

The second concern is that the two procedures are testing different things. A good description of this is in David Hand's great paper "Deconstructing Statistical Questions". This is his example 4 (http://statlab.bio5.org/sites/default/files/fall2014/hand-deconstructin.pdf). His point is that the two approaches are addressing DIFFERENT research questions. If the researcher doesn't know which they are asking, Hand's argument is that they should figure that out before doing the analyses (or before designing the study). At least wrt causal inference, Wainer, Pearl, etc., discuss when the ANCOVA approach is more likely to be valid.

I think it is a slippery slope running multiple hypothesis tests for different research questions and then selecting "the most powerful" (which I assume means lowest adjusted p value here), and having that determine what your research question is. I would think the research question should be known beforehand, and not determined by p values (Gelman and Loken make similar arguments in their garden of forked paths paper).

Again, I might be missing something, so correct me if I have read your advice incorrectly.

here is an example where the two approaches would lead to the opposite conclusions if it were believed that they were addressing the same question. London and I discuss how each approach addresses a different questions so each is valid in some situations.

Chapter Analyzing change between two or more groups: ANOVA versus ANCOVA

One important consideration is that between-participant variables such as age are individual differences that are part of the repeated measures subjects term. So simply adding a covariate like age decreases the subject term and has no impact on the tests of other effects.

For variables like to impact on the model you need to also include interactions of age by time and by group (which is probably a sensible move).

However, using your time 1 measure as a covariate fundamentally changes the model (generally to a more plausible one - but look at the references Dan provided).

So in this case there is an option to have three independent measures ANOVA models: age x time x group or a repeated measures model with age x time x group x social_cognition. (I personally would avoid MANCOVA as it can be uninterpretable and people follow it up with the univariate analyses anyway).

In any case you should centre the covariates before including them in a repeated measures model or MANCOVA. Some software will not handle uncentred covariates at all sensibly.

I think what James means is that there are multiple ways to model the situation. How you model it determines the assumptions so the better way is really determined by which set of Assumptions you can satisfy. I'd recommend Neil Timm, Applied Multivariate Analysis, Springer.