Are the 3 IV measures from the same measuring instrument?

What is your missing data situation?

If not I would perform a repeated measures ANCOVA separately for each of the 3 IV with the 2 IV you wish to control for as the covariates.

Are you using SPSS, SAS or STATA?

If so, look into a GEE or Mixed Models procedure.

No, it wouldn't generally be appropriate to collapse across two control IVs based on non-significant t-tests with their marginals. Those t-tests would only detect main effects of the control IVs, not any interactions they might have with each other or with the other IVs. Presumably, "controlling for" these IVs means taking into account _all_ effects they might have on the results (whether through main effects or through interactions), so an analysis ignoring the interactions provides an extremely limited sort of control. The full 5-way ANOVA will give you separate tests of all the main effects and interactions, so I think you should do that.

I don't see any use for ANCOVA here, because (a) all factors have 2 levels, and (b) the ANCOVA would also ignore any interactions involving the control IVs (if you set up the ANCOVA model in the usual way).

You don't really say why you want to avoid the 5-way ANOVA in the first place, so maybe my answer doesn't fully address your core concerns. If you are worried that the more complicated 5-way ANOVA somehow degrades the information you get about the 3 IVs of main interest—don’t be. Because of the fact that this is an RM design, you will get exactly the same F’s (and dfs) for these 3 IVs and their interactions within the 5-way ANOVA as you would in a 3-way ANOVA where you have averaged scores over the 4 combinations of the two control IVs. (Doubtful? Try it!)

Or maybe you are worried about the larger ANOVA-wise Type I error rate associated with the 5-way ANOVA (31 F’s, each providing an opportunity for a Type I error) as compared with the 3-way ANOVA (only 7 F’s)? That is a legitimate concern, but it is simply “the cost of doing business”. There is really there is no way around this problem except to ignore information, which is a cure that is worse than the disease. To take it to an extreme: If you really want to minimize your lifetime Type I error rate, you should stop doing experiments where you test some Ho that might possibly be true.

I think you should integrate your IVs in the model with the other IVs variable because testing each IVs independently to each other may induce error of interpretation. Indeed it's possible that one variable is no significant when she is integrate alone in the model whereas she becomes significant when she is integrate with other variables.

Thank you Raymond, Jeff, and Mohamed for your insights. Raymond, I am using SPSS, and have not yet figured out how to set up within-group covariates in the GLM repeated measures window (between-group covariates seem easy enough to configure). As per SPSS "wide format", each of those 32 levels has its own column in SPSS. Any ideas?

Jeff, I appreciate your input, in particular your third paragraph has identified and addressed the fear that I had, that there would be some loss of information associated with a full 5-way ANOVA. Given what you have said I will steer clear from the omnibus t-tests idea.

What if I had a second experiment (E2) that utilised the same stimuli and basic experimental procedure as the first (E1), but tested different IVs... when analysing the results of E2, could I collapse the variables that E1 told me produced non-significant effects?

Regarding your follow-up question about E2: No. Power is never 100%, so the null results in E1 do not prove that the control IVs have no main effects or interactions. Again, I don’t see a theoretical advantage in collapsing (though it might simplify the calculations), and the disadvantage is that you lose the possibility of identifying real effects involving the collapsed IVs.

I would say no - this is generally a bad idea, for the reasons outlines above. If your having trouble with SPSS's repeated measures ANOVA window not giving you the options you want then perhaps an alternative approach might be the way to go. Maybe regression if your more familiar with that (it's all the GLM after all).

As for the second question - again there is always the possibility of interactions in E2 that are not there in E1 (or even main effects?). The new IV's might be more sensitive to the effects measure by the control conditions (or worse the presence of the various stimuli interact in such a way to produce context dependent effects).

I think here however the question is really about the theoretical question your testing not the stats. There must be a reason why you included the control conditions and presumably that reason is still valid. Hence the comparison to the control conditions is still necessary to show an effect that is different to the controls. One can use specific contrasts to test the experimental conditions against each control condition or if it is theoretically justified, the combined control conditions.

The important issue is to address the research question and inferential stats are just to help you make statements about the probability of observed outcomes.

I will happy support the answer from Jeff Miller; statistically you are gaining nothing by pretending that you don't have information when you actually have it. Worse, the effects of those "control IVs" not being precisely defined means that you are as likely to hide something as to clarify it.

That being said, what would determine how I would first set up this design and then report it are my hypotheses. From your original description is seems that you wished to establish the null effect of these control variables. If you have hypothesized that they have no impact then you will be best off showing that they have no main effects and no interactions within the power of the design. If you have hypothesized positive effects for other variables then you ought to be satisfied by the overall anova. I think that the criticism about the overall alpha inflation is much overblown when you reach the level of specificity that seem implicit in your design - 3 positive main effects and 2 null main effects that you can predict in advance are probably are not due to chance.

Now, if you have less to say about the possible interactions you will have some choices in what you wish to report. That is, those interactions that involve variables on interest but that are not specified in your theory I would tend to just go ahead and report. The 4th and 5th level possible interactions are probably underpowered and so you could consider not testing them. Not having any idea about the number of observations at issue I will just say that I have never been tempted to conduct omnibus tests in a 5 way anova. I would be far more likely to build as specific a set of contrasts as possible to test my theory.

Good luck.

Thank you all for sharing your interesting and helpful ideas. I have conducted the 5-way ANOVA, and have found support for my three experimental hypotheses. Further, one of the two null hypotheses (i.e. relating to the control IVs) has been challenged by a significant main effect; the IV in question is further embroiled in a first- and second-order interaction! So Bruce, it looks like I can anticipate reporting more than I had hoped to - thankfully there are no 5th level interactions.

In the follow-up experiment (E2), participants are measured across similar conditions as in E1, but two signal detection outcomes are measured: sensitivity (d`) and response criterion (c). My hunch is that this design will entail a repeated measures MANOVA, but I am curious: Is it theoretically possible to run a separate alpha-adjusted ANOVA for each DV? Thanks in advance for your advice.

It depends a little on your research questions, but with d' and c, I would normally expect to see separate non-alpha-adjusted ANOVAs for each of these two DVs, because they would theoretically be sensitive to two such different types of effects. Arguably, you would want to retain your full Type I error rate for each of the two different effects, so as to preserve whatever power you have to detect each of the effects.

The justification of a MANOVA usually rests on the premise of some relationship between your measures. In the case of d' and c, you may well have an argument that combines them; that would lend itself to a multivariate assessment. You could also force an explicit relationship by estimating A'. This may be the best option if your hypotheses can be expressed in this form.

Very often reviewers will ask for the 2 variables to be analyzed separately even after a multivariate analysis was conducted. What you have asked is whether it is possible to run these 2 ANOVAs with some adjustment to alpha.

My answer would be yes - you could. But I would then hope for reassurance on two points. The first would be about how independent the measures really are in your context. It may very well be the case that it is far more difficult to justify a particular approach to correction than it was to justify the analysis in the first place. Second, are you doing this post-hoc? Because the power issue that Jeff Miller has brought up would be far more important in most designs.

Given the theoretical possibility that this could and occasionally should be done, why are you so concerned with Type 1 error rates? If you are investigating an effect whose existence was already established in a first experiment then you are far out of the range in which anyone would argue for a true null hypothesis. Second, if you can again specify an entire theory across the available predictors and measures then you can generate a very detailed hypothesis above and beyond what is offered in some series of omnibus F tests. Remember, the statistical model is only loosely analogous to reality. The validity and fidelity of your conclusions rests in the logic of the experimental situation.

"...2 IVs that I want to control for and show that they have no bearing on the main results"

There is simply no way of showing that an IV has no effect on a DV or that is does not have an interactive effect on the DV via another IV... within the Fisher(-)ian framework.

To sum up, within the statistics most psychologist use, and introduced by Fisher at the beginning of the 20th century, you start from the hypothesis that *there is no effect of a variable*, aka the null hypothesis, and you work your way towards (possibly) showing that the probability of having the data you actually have when considering that the null hypothesis is right is too low. In other words, you start with a model (the no difference model, aka the null hypothesis model, also noted H0) and you "compute" the probability of having the data you actually have given your model. This is noted p(data|model H0). This bit of notation just to make it obvious that within the Fisherian statistics framework you start from a model that says that the IV under scrutiny does NOT have an effect. So, with this as a starting point, you are limited to either rejecting H0 (thus establishing that your IV in fact does have an effect on the DV) or ... well, nothing else. This framework leaves no place for another conclusion, I mean, you cannot come to the conclusion that the IV under scrutiny does NOT have an effect, since this is the starting point of the Fisherian hypothesis testing reasoning.

Happily though, you can affirm that an IV does not have a direct or indirect ( and by indirect I mean through an interaction) influence on a DV ... IF you consider the Bayesian statistics framework. Within this quite different (and mistakenly thought as difficult) framework, you do not "compute" the probability of having the data you actually have given your model, but just the other way around: you compute the probability that your model is the good one given the data at hand, that is p(model | data). So you consider all the possible models (for instance, a model that includes the IV you want to show does not affect the DV, and one that does not include it) and you get for each model the probability that that specific model if the right model. With mode than one IV (and of course even more so with 5 IV) the number of models to compare can be a little high, but if you keep the number of IV within the reasonable (and 5 is reasonable) then this is very manageable. The magic of it is that you can affirm a model that "says" such or such IV does not influence the DV, i.e. affirm a null hypothesis (yes, that's right, the thing your statistics teacher always told you that you ought never say).

Let me illustrate this for you with a very simple (simplistic) example. If your IV is called IV and your DV is DV, then you would have (with the classic notation adopted in glm in R) two models to oppose one to another:

M1 : DV ~ IV (this is the model that says in fact DV = b0+b1*IV)

and

M0 : DV ~ 1 (this is the model that...

Looks like the end of my previous comment is corrupted, so here it is:

...

and

M0 : DV ~ 1 (this is the model that says in fact DV = b0)

The software would then give you p(M0|data) and p(M1|data) [or sometimes only the BIC for M0 and the BIC for M1]. At this point, if p(M0|data) > p(M1|data) [alternatively if the BIC that corresponds to M0 is smaller than the BIC that corresponds to M1] then you can affirm that M0 is the model that is the most probably true, that is, you can affirm the null hypothesis.

With a within-subject IV, the syntax is different, but the reasoning is the same.

Hope this helps (a little) and makes you curious about the marvellous world of Bayesian statistics ;)

Serban, I really appreciate (and enjoyed reading) your detailed post. I have had an interest in Bayesian interpretations for a couple of years now, and think I understand the conceptual basics of it. However, I have yet to tackle my own data with a fully Bayesian approach, as I am still trying to grasp the finer points. I am familiar with SPSS but not R, so I am sorry to say I got a bit lost reading your notation.

In my PhD experiments, one thing I am manipulating is the probability of target vs. non-target stimuli in a given block. Without being told what this probability is, participants must at each trial make a choice between two responses: 'target present' and 'target absent'. In 'the real world', this probability is about .5, so I am assuming that that is the model participants will have adopted prior to the experiment commencing. The trick will be quantifying the extent to which participants' prior beliefs remain fixed or change as a function of changes to the experimental prior.

Justin,

Thanks for your interest and your kind reply.

I understand what you are saying, but I was not going down that path in my answer. My thinking was along a model comparison approach. Sorry if my post was not clear enough, I'll try to make this clearer here.

For clarity sake, let's simplify your experimental plan and include only the DV and 2 IV, presence_of_target and, say color.

Here are ALL the possible models with these 2 IVs (I give them names of the kind M1, M2 and so on to be able to refer to a particular model later):

M1: DV = b0+b1*presence_of_target+b2*color+b3*presence_of_target*color+random_error

M2: DV = b0+b1*presence_of_target+b2*color+random_error

M3: DV = b0+b1*presence_of_target+random_error

M4: DV = b0+b1*color+random_error+random_error

M5: DV = b0+random_error

[Of course, b0 (for instance, and the same goes for the other b's and the random_error term) is not the same from a model to another, I should have used subscripts but I'll avoid these for concision sake.]

Now, these 5 models are all the models possible with 1 DV and 2 IV.

The Bayesian approach consists in estimating how well each of these 5 models fits to the data, or better said, in finding which of all the possible models (here, five models) best fits the data. This is given by the probability p(model |data), and more often than not by a number, the BIC (it's not totally correct to say this, but to give you a grasp, you may think of the BIC as a "distance" to the "perfect fit", so it is the model which has the lowest BIC that is the best).

This is about all you need to know. No prior belief stuff, you'll have no calculation to do yourself, the program does it for you. It's very much like SPSS if you like, you feed the data and get your answer. The only difference is that with this Bayesian approach it is YOU who specifies the models to be tested, so you must take care to consider ALL the models there are when looking for the best model.

You can install very easily R on your computer from here (http://cran.r-project.org/), then install a visual interface "package" called R2STATS to carry out your analyses.

A last thing is how to interpret the results. Well, imagine M1 has the lowest BIC ; in this case, presence_of_target DOES influence the DV (though not in the same way for the different levels of your other IV, color in my example). If it is M2 the model with the lowest BIC, then you have a main effect of presence_of_target and a main effect of color (and no interactive effect of presence_of_target and color). If it's M3 the model with the lowest BIC, then you have a main effect of presence_of_target and nothing else (no color main effect, no interactive effect of presence_of_target and color). If it's M4 the model with the lowest BIC, then you have a main effect of color and nothing else. If it's M5 the model with the lowest BIC, then there is no effect whatsoever (neither of presence_of_target nor of color). I think you get it.

Let me know if you...

Again, it looks like the end of my previous comment is corrupted, so here it is:

...

Let me know if you want to follow this track and if I can help you.

Cheers,

SCM

Thank you for the informative post Serban, the way you have described this method of model testing makes sense to me. When I have some time I will check out R and have a play with some data. Your posts have inspired me to read about BIC, a term which I was unfamiliar with until you mentioned it. Like ROC curves, BIC is something I would like to compute but need to invest more time into studying. Cheers for the leads!

No problem, hope this helps.

Cheers :)