You say "matched groups," so I assume that a contrast between two matched groups creates a within Ss design. As well, I am guessing that the contrast between these groups was on what appears to be 6 DVs.

In general, yes, most methodologists would deem the likelihood of falsely rejecting a null to be inflated beyond whatever alpha was used, unless some sort of steps are taken to control the situation. A simple Bonferroni correction is of course possible, and in my view better than ignoring the problem, but there are other possibilities (Holm, FDR, etc.). As well, the Bonferroni correct probabilities need not be equal for all nulls -- they just need to sum to alpha.

A first impression, is that yes the N of 20 is not impressive, but the way to look at the sample size is in respect to power. I don't have enough information regarding that, as you would need to specify an alpha, and effect size you wish to identify to calculate power. For instance, if I assume alpha=.05, and the effect size you wish to identify is .5, with an N of 20 the power is < .60. Not too impressive, but if there is to be a per-hypothesis error correction, say simple Bonferroni, then the alpha changes to around .008 (.05/6), and power takes a drastic hit at < .3.

Best wishes in your research.

John

Thank you so much John, perfectly explained!

Best wishes from Natalie

You don't say what statistics were performed. For example, were they testing if the median correlations for these two groups were the same, if the overall means were the same, or something else? In general, if there is a family of tests/p-values, using something like Holm's adjusts for type I error rate but can cause problems for Type II rates, so it depends how "significance" is used (and the costs of different errors).

On sample size, again, is depends how these data were used.

Happy to help.

In general, adjustment for multiple hypothesis testing (probability pyramiding) will always cost power, unless we ignore the problem; I don't advocate that. In my opinion, that is as it should be; parsimony in selection of DVs is an important design consideration. You will (and should) pay for a shotgun approach to selection of DVs; consider their selection carefully. I sometimes ask my students to envision putting a power coin into a DV vending machine -- each costs power.

That's a brilliant analogy, the idea of the power coin in the DV machine! Very, very helpful and imaginative!

Aw shucks -- thanks:-) Analogies do seem to work well for my students.