I understand that a sample size to detect an interaction is 4x higher than to detect a main effect association. Thus, for a given, fixed sample, is it valid to raise the alfa in order to detect potential interactions? If so, what P-value can be considered significant in interaction term analyses?
Hi Jochen, did not notice I wrote that much, no? To make myself more understandable I rephrase my 10 words...From P=0.25 potential interaction can be considered as significant. Does not mean it is always the case, as you point out, further analysis might start from there. The graphical analysis can help too.
I agree with Christian. If you are interested in the interaction terms, then you need to plan sample sizes accordingly. You also need to plan your degrees of freedom, making sure that you account for differences between categorical and continuous independent variables in terms of the required degrees of freedom.
Completely agree with the need to plan a sample size accordingly; although this is not possible for some projects especially those based on existing databases (which is my case).. Thoughts?
Hi,
Which P-value you refer to? Interactions are usually considered significant from P =0.25
Sorry Luc, but you seem to contradict yourself in your post. Firts you 'suggest to not only apply "p-value stats", but also fit specific models, based on a functional understanding of your data/system' - I very much agree to this point - and later you write that 'you should remove higher order terms when they are non-signiifcant until only significant higher order and/or main effect terms remain' what is exactly opposed to your former statement. Selecting models based in p-values is a very bad strategy.
One key problem of Thais is that the 0.05-limit for "significance" is not sensibly defined. There is no reason to chose this limit. The only only only sensible way to define such a limit is by quantifying the expected losses for type-I and type-II errors and then select alpha and beta to get a decision staregy minimizing the maximum expected loss. Without having a clear and reasonable quantification of such losses (i.e. a defined loss function) the whole procedure is logically flawed anyway.
Especially in exploratory research the loss acssociated with a type-II error might be much more severe than that associated with the type-I error. Who knows? And if one doesn't know - what's the point of doing hypothesis tests anyway?
One may argue that a significance test ala Fisher is ok, but that is mere window dressing. Fisher insited that the p-value is judged based on the whole context (and using a "general" or "common" cut-off like 0.05 is clearly a stupid thing to do in this philosophy!). Unfortunately, there is no help on how to do this, and Fisher failed to give any useful advice.
Not all databases are suitable for answering all questions. If the data were not gathered in a manner appropriate for the analysis that you want to do, your only choice is to change what you want to do, or change how you are doing it. You can stuff numbers into any model, and you will get numbers back out. It doesn't mean the activity is worth doing.
As Jochen points out, 0.05 is just a number. There is nothing magical or divine about it. What you need to do is find a different number that you can justify with something other than "I used 0.32 as my cut-off because that gave me significant differences that made me happy." Can you use maximum likelihood methods or Bayesian statistics to help? Is there some external factor/information that can be used? Is there some other study indicating that a difference of "this magnitude" has a significant biological consequence?
Hi Jochen, did not notice I wrote that much, no? To make myself more understandable I rephrase my 10 words...From P=0.25 potential interaction can be considered as significant. Does not mean it is always the case, as you point out, further analysis might start from there. The graphical analysis can help too.
Hi Luc, perfecty agree with you. I would add that there also other factors to consider apart from the simple p-value of an interaction: two examples in clinical research are biological plausibility and previous evidence in literature.
It looks several papers support, do not raise p-value for testing an interaction.
such as
https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0071079&type=printable
Anyone could point out any reference in pubmed regarding this topic? I mean, any paper discussing the topic of potential cutoff for p values for considering "interesting" or "valuable of further investigations" such formally not significant interactions??
Thank you for this question, I think that is still fascinating that in 2021 I can find this question and the discussion. First we need to think on the meaning of a P value, which is arbitrary and was "set" as the standard gold before computers were around. So a P value of 0.055 does not mean that the is a difference? I always rise my P value for an interaction to P < 0.1, but in the manuscript I add the actual P value and let the reader to get her/his own interpretation. How much risk is the reader going to accept? That is up to each reader.
- Badges
- Science topic