There are (at least) two schools of thought and (at least) the two ways of proceeding that you have noted. The question of whether or not to include nonsignificant effects that have theoretical (e.g., biological) importance has to do, fortunately or unfortunately, with the importance of the effects to the domain under study.  My own preference is to include what is theoretically important, and the interesting question for me is whether anything novel revealed by the experiment is significant or contributing a large effect estimate even when traditionally important variables are in the same model. And besides that, significance of pre-existing theoretically important variables is cause for fleeting reassurance, but their possible nonsignificance is entirely uninterpretable and harmless one way or another. Null effects by themselves provide no new information that requires/invites a conclusion.

When in doubt, just comply with the reviewer and see what happens.  If they are just asking for p-values, just provide p-values for what you had submitted.  If reviewers are just asking for p-values and then get a revised version of the manuscript with much more than p-values added, it risks making them more suspicious. They don't want a moving target so much as they want to see clear incorporation of their feedback.

Best wishes,

Damian

Dear L. Thompson. I recommend you to take a look in two helpful references:

Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR, Stevens MH, White JS (2009) Generalized linear mixed models: a practical guide for ecology and evolution. Trends in Ecology and Evolution 24 (3):127-135.

Zuur AF, Ieno EN, Walker NJ, Saveliev AA, Smith GM (2009) Mixed effects models and extensions in Ecology with R. Springer, New York.

To test the significance of variables is an important to model selection, but some best models may contain non-significant variables. Thus, it's not necessary to remove non-significant variables. After the step of model selection, you should to check model assumptions. Normality of residuals, fitted residuals x predicted residuals, etc. AIC-based selection could be sensitive to these 'problems'.

Best,

Davi Tavares

You should report both (effect size and p-values). In particular, it doesn't make sense to report effect size if the effects are not statistically significant... so, you should provide the reviewer with the complete information so she/he can judge your model (and results) better.

Thanks everyone, I'll go with the reviewers' request for p-values next time, but I'm relieved to hear there are a number of ways to do it, depending on the situation. 

Glad you feel relieved, Lindy!  It can be very stressful, for sure. There are few morals in statistics--there are rather very many well-informed opinions that often conflict.

I did want to clarify that it can actually make sense to report effect size even for statistically non-significant. In fact, many proponents of effect sizes refuse to deal with p-values at all. There are several great arguments *against* p values and null-hypothesis significance testing and *for* more Bayesian modeling that focuses more on estimating the "real" effects whether or not they stand up to an, at times, artificial threshold.

However, even without going to the Bayesian side, the best reason to report effect sizes even for non-significant is that *both* effect size and non-significant p values are important information for doing power analyses and for proposing judicious sample sizes and treatment for grants and clinical trials. Everything we hide away falls under the looming "file-drawer problem" of failed results failing without anyone but the scientists have the benefit of the knowledge.

Best wishes,

Damian

I basically agree with the above, but the statement "but their possible nonsignificance is entirely uninterpretable" strikes me as too strong - usually yes, but not always. If I look at a confidence interval for an odds ratio of effect X and it  is [0.98, 1.02] and other odds ratio for the outcome are, for example 2 and 3 and 0.33,  then my interpretation is that there is evidence that X does not strongly effect the outcome. I certainly agree that confidence intervals are required to claim anything like that and that a non-significant p, by itself, is mute.

Lindy -

Your very relevant question is related to a similar discussion at the first link below, where I think you will find interesting answers, especially those by Theo Dijkstra. There was quite a consensus, in my opinion, that as noted above, the subject matter theory is very important.

As for p-values, the notes at the second attachment regard how problematic is the use of a single p-value, as it is a function of sample size. At a given level, a small sample size will tend not to 'reject' a given hypothesis, no matter how far from true it may be, whereas a large sample size will tend to reject a hypothesis which is nearly true. Confidence intervals are generally of more practical value. I think more people are realizing this now, though it has become traditional to ask for p-values. I recall one answer to a question on RG where it was noted that p-values were not meant to be anything but a guide along the way during research. They seem to have taken on a life of their own, so to speak, I'd say.

Best wishes - Jim

https://www.researchgate.net/post/When_doing_multiple_regression_analysis_should_I_put_all_predictors_in_regression_model_or_only_those_having_a_significant_correlation_with_outcome

Article Practical Interpretation of Hypothesis Tests - letter to the...