Was wondering if it is possible to run a non-parametric permutation test (preferably cluster corrected) for an anova design. I'm currently trying to find the best method for deriving the statistics from some data and the comparisons between groups will be as follows:
- i have 3 groups
- each group completed 2 conditions (perceptual outcome 1, perceptual outcome 2), in the same session.
- would like to compare any effects of group on the amplitude differences in the perceptual outcomes
Is this possible? I'm not so sure about whether or not the test would be sensitive to using an f value in place of a t value. I've read through the Cohen book, and the Oostenveld papers and the theory behind it all makes sense, I'm just unsure of how to tweak this in practice.
Ideally I would like to avoid using fieldtrip to run this, as A) I'm not a fan of how it works; B) I don't have the time to learn another toolbox.
Any suggestions are greatly appreciated. Thanks
Hi Nik,
Yes, it is possible to run such a test, all you need to do is use the F-ratio(s) instead of t-values. In your case, it seems that you have a 3 (groups: between subjects) x 2 (conditions: within subjects) factorial design, and that you are looking for the interaction effect.
So what you need to do is fit a 3x2 between-by-within subjects ANOVA for every channel and every time point, and get the three F-ratios (the 2 main effects and the interaction).
Then permute the data - and here you may have to ask yourself how you are going to do that, since you probably do not want to permute freely, as you have one between subject factor (group) - and repeat. In the end, you should have a reference permutation F-ratio distribution for every channel and time point.
For the cluster correction, there are different ways in which you can do this. One is the Maris and Oostenveld 2007 way (spatio-temporal). Another, and perhaps simpler way is to sum all the F-ratios across channels for every time point (this is the equivalent of the Tsum2 measure initially proposed by Blair & Karniski, 1993), and do the temporal cluster correction a la Maris and Oostenveld 2007 (for single channels).
There are also other multiple comparison methods you may want to try. The only published paper that I know that goes in some detail on these issues for higher order designs like a factorial anova is:
Wheldon, M. C., Anderson, M. J., & Johnson, B. W. (2007). Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons. Australian & New Zealand Journal of Statistics, 49(4), 397-413.
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-842X.2007.00491.x/full
As for how to implement this, I'm not sure if there are any available packages where this particular permutation test is implemented. You may have to code it yourself. I do have some R code to do a 2 x 2 ANOVA with temporal cluster correction, but it won't work on your 3 x 2 design (it only works for 2 x 2 fully within subjects design).
References:
Blair, R. C., & Karniski, W. (1993). An alternative method for significance testing of waveform difference potentials. Psychophysiology, 30(5), 518-524.
Karniski, W., Blair, R. C., & Snider, A. D. (1994). An exact statistical method for comparing topographic maps, with any number of subjects and electrodes. Brain Topography, 6(3), 203-210.
Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG-and MEG-data. Journal of neuroscience methods, 164(1), 177-190.
Wheldon, M. C., Anderson, M. J., & Johnson, B. W. (2007). Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons. Australian & New Zealand Journal of Statistics, 49(4), 397-413.
Hi Nik,
Yes, it is possible to run such a test, all you need to do is use the F-ratio(s) instead of t-values. In your case, it seems that you have a 3 (groups: between subjects) x 2 (conditions: within subjects) factorial design, and that you are looking for the interaction effect.
So what you need to do is fit a 3x2 between-by-within subjects ANOVA for every channel and every time point, and get the three F-ratios (the 2 main effects and the interaction).
Then permute the data - and here you may have to ask yourself how you are going to do that, since you probably do not want to permute freely, as you have one between subject factor (group) - and repeat. In the end, you should have a reference permutation F-ratio distribution for every channel and time point.
For the cluster correction, there are different ways in which you can do this. One is the Maris and Oostenveld 2007 way (spatio-temporal). Another, and perhaps simpler way is to sum all the F-ratios across channels for every time point (this is the equivalent of the Tsum2 measure initially proposed by Blair & Karniski, 1993), and do the temporal cluster correction a la Maris and Oostenveld 2007 (for single channels).
There are also other multiple comparison methods you may want to try. The only published paper that I know that goes in some detail on these issues for higher order designs like a factorial anova is:
Wheldon, M. C., Anderson, M. J., & Johnson, B. W. (2007). Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons. Australian & New Zealand Journal of Statistics, 49(4), 397-413.
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-842X.2007.00491.x/full
As for how to implement this, I'm not sure if there are any available packages where this particular permutation test is implemented. You may have to code it yourself. I do have some R code to do a 2 x 2 ANOVA with temporal cluster correction, but it won't work on your 3 x 2 design (it only works for 2 x 2 fully within subjects design).
References:
Blair, R. C., & Karniski, W. (1993). An alternative method for significance testing of waveform difference potentials. Psychophysiology, 30(5), 518-524.
Karniski, W., Blair, R. C., & Snider, A. D. (1994). An exact statistical method for comparing topographic maps, with any number of subjects and electrodes. Brain Topography, 6(3), 203-210.
Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG-and MEG-data. Journal of neuroscience methods, 164(1), 177-190.
Wheldon, M. C., Anderson, M. J., & Johnson, B. W. (2007). Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons. Australian & New Zealand Journal of Statistics, 49(4), 397-413.
Hi i think that you can do this and the previous response is an excellent overview of important considerations.
However i also have different suggestion:
One of the big problems in anova is full filling the statistical assumptions of your test. I would also think of centring your data in order to aline the different scales of your outcomes (assuming that they are different). That would take care of any collinearity and also check the assumption of normality (you could also considering to center your data by z-score.
You could also use linear regression if your interessted in group effects on the amplitude of outcomes. By linear regression you can really determine the contribution of group allocation to the amplitude. This is more robust in small sample sizes.
Good luck and happy number crunching:)
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics