Hello,
I have a 3*3*2*2*2*2*3*3= 3^4*2^4 (i.e. Total eight factors: four factors with two levels each, and four factors with three levels each). To get a small effect size in regression, the sample size needed for a full factorial design is over 8,500. I am only interested in the main effects of each factor and the two-way interactions (actually only some of them). Thus, the fractional factorial design allows me to test all main effects and all two-way interactions with less sample size needed. Would you please give me some advice about how to make it happen?
1. From this material (https://www.itl.nist.gov/div898/handbook/pri/section3/pri33a.htm), I know I can convert one three levels factors into two two levels factors. Does that I mean I have a (2*2)^4*2^4=2^12 full fractional design? After converting, I believe I am also interested in some three-way interactions because some variables are split.
2. To decide which high-way interactions can be compounded into which main effect or low-way interaction, should I use a sign table? ( I don't have my data yet.)
3. When using software to run regressions, we can examine only some of the main effects and interactions by simply indicating them in the command. Do I need a smaller sample size if I will indicate/ examine only some rather than all effects( i.e. partial comparisons) in the regression command? How does it different from a fractional factorial design?
Thank you for your time and help in advance!
Hello Qiaoyuan,
That's a lot of variables! The short answer is, yes: you can opt to suppress any or all higher-order interactions in a model. Note that doing so is tantamount to presuming that they are nil in the population. Whether you do or don't invoke a full factorial sampling doesn't matter with respect to the decision (though it's true that a lot of stat texts will show designs in which incomplete sampling is applied, usually as a way to economize on sample size).
If your research questions extend principally to main effects, then a priori sample size determination should apply to just the ability of those tests to detect your minimally important effect size.
It really makes no difference whether you fit a regression model or use anova with suppressed interaction terms (they're the same creature underneath it all: GLM).
Good luck with your work.
Hello Qiaoyuan,
That's a lot of variables! The short answer is, yes: you can opt to suppress any or all higher-order interactions in a model. Note that doing so is tantamount to presuming that they are nil in the population. Whether you do or don't invoke a full factorial sampling doesn't matter with respect to the decision (though it's true that a lot of stat texts will show designs in which incomplete sampling is applied, usually as a way to economize on sample size).
If your research questions extend principally to main effects, then a priori sample size determination should apply to just the ability of those tests to detect your minimally important effect size.
It really makes no difference whether you fit a regression model or use anova with suppressed interaction terms (they're the same creature underneath it all: GLM).
Good luck with your work.
David Morse Hello David,
Thank you very much for your extremely helpful answer.
If I am interested in the main effects and two way interactions( e.g. Number of tested predictors= 73)only, in the priori sample size determination, is the total number of predictors also 73? Or should I include the number of predictors of all high-way interactions in the total number of predictors?
Thank you again for your time and help.
This does not answer you original question! But considers an exploratory way of modelling a situation with lots of interactions that tries to take account of small numbers
Article Uncovering interactions in multivariate contingency tables: ...
"The procedure produces precision-weighted estimates of the observed:expected rates for each and every cell, together with associated Bayesian credible intervals, and is illustrated using a large survey data set relating voting (and abstaining)at the 2015 UK general election to age, sex and educational qualifications. Crucially while fine detail can be explored in the analysis unreliable rates for particular subgroups are automatically down-weighted to what is happening generally. The identification of reliable differential rates then allows a simpler hybrid model that captures the main trends to be fitted and interpreted. "
and a follow up
Article Using Shrinkage in Multilevel Models to Understand Intersect...
" The hope is intersections appearing statistically significant by chance in a fixed-effects regression will not appear so in a multilevel model. However, this requires assumptions that are likely to be broken. We use simulations to show the effect of breaking these assumptions: when there are true main effects/interactions, un-modeled in the fixed part of the model. We show,whilst the multilevel approach outperforms the fixed-effects approach, shrinkage is less than is desired, and some intersectional effects are likely to appear erroneously statistically significant by chance. We conclude with advice to make this promising method work robustly".
Hello again Qiaoyuan,
I presume you're asking in the context of using a power analysis program, such as G*Power. The answer depends on the hypothesis(es) of interest. If your principal hypothesis is that NONE of the independent variables influences scores on the DV/s, then that's akin to testing that R-squared = 0 for a regression model, and you would have however many IVs in the model as you plan.
On the other hand, if you're more interested in individual IVs (e.g,, "Does factor A make a difference in scores, holding all others constant?") then the number of IVs is one (at a time), with k levels. So, k is the number of "groups" and k - 1 is the df for that main effect or factor.
Good luck with your work.
Kelvyn Jones Dear Kelvyn, thank you very much for sharing the articles. I learn a lot about exploratory study from them.
Solution to this is of interest to me too. Yes, exploratory approach and the fractional factorial design which allows testing all main effects and all two-way interactions with less sample size. Get your data pls, decide which interaction effect confound the others in the fractional design ,... am waiting to read "it worked" and your research questions have satisfactory answers.
This is a rather complicated design and the NIST handbook is not regularly updated. Because of that, I would suggest in addition, this standard work in Design of Experiments: https://b-ok.cc/book/3591248/4df2ac
Best wishes, David Booth
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size