Choosing whether or not to model a factor as random is a decision that should be made based on the assumptions you are willing to make regarding your data and statistical model, rather than simple statistical significance per se (though you haven't indicated exactly HOW you determined significance in this case, but I'll get back to that thought in a moment).

First, you have to think about whether or not it makes sense to model a factor as random. Ask yourself the following questions: are you interested in quantifying the degree of variation attributable to the levels of that factor (e.g. how much of the variation in your outcome variable is related to the block or species identity)? If yes, then the factor should be treated as random.

Are you including these factors in the model because you want to directly compare the means (e.g. compare species 1 with species 2) or simply because you are trying to account for the variability across species? If the latter, the factor should be treated as random.

Are you interested in inference on these specific factors, or do these factors represent a subset of a wider population to which you want to generalize your results? That is, are you interested specifically in the species in your study, or do these species simply represent a subset of a broader phyla you are interested in? If the latter, the factor should be treated as random.

Note that these are all separate questions from any statistical output, and in fact should be decided a priori.

All of that said, depending on your precise research questions, there are situations in which it is reasonable to "change your mind," and treat a factor as fixed instead of random or to remove it from the model entirely. For example, it may be reasonable to remove the "block" effect in your model if you have strong evidence that it doesn't have a strong relationship with the outcome of interest, in which case it is more parsimonious to remove it.

However, you have to be careful what criterion you use to decide what constitutes this evidence. This is where we get  back to the question of what exactly you mean by "not significant". How did you measure this significance? The most appropriate way to decide whether you can remove the effect from the model is to perform a likelihood ratio test between a model including the random factor and a model excluding the random factor, and test the significance of that test against a Chi square distribution.

Factors that are not significant should not be part of the model. BUT take care: "significance" is not simply judged from the p-value (and a comparison to some arbitrary cut-off like 0.05)! Significance is judged by expertise, depending on the structure and the aim of the study in the context of expected consequences of a decision (e.g. of whether or not to keep the variable in the model).

All this can be done before the study, leading to a definition of acceptable error-rates and consequently to a corresponding specification of alpha and beta,what eventually gives a rejection region of the test statistic. The decision is then taken simply based on the fact whether or not the sample statistic falls into this rejection region. -> "Neyman", "hypothesis test"

If the data is "just there", and there exists no specific beta and no power analysis, one has to think about all these things after the experiment. A sensible control of error rates is not possible. The best option is to judge the observed (estimated) effect size and the apparent precision of this estimate, again depending on the structure and aim of the study and the scientific context. (that's a bit like Fisher used his "significance tests", but after all his way was not logically consistent)

Interactions: again you should judge the significance of the interactions you observe. If they are considered significant then, for sure, you should not omit them.

Choosing whether or not to model a factor as random is a decision that should be made based on the assumptions you are willing to make regarding your data and statistical model, rather than simple statistical significance per se (though you haven't indicated exactly HOW you determined significance in this case, but I'll get back to that thought in a moment).

First, you have to think about whether or not it makes sense to model a factor as random. Ask yourself the following questions: are you interested in quantifying the degree of variation attributable to the levels of that factor (e.g. how much of the variation in your outcome variable is related to the block or species identity)? If yes, then the factor should be treated as random.

Are you including these factors in the model because you want to directly compare the means (e.g. compare species 1 with species 2) or simply because you are trying to account for the variability across species? If the latter, the factor should be treated as random.

Are you interested in inference on these specific factors, or do these factors represent a subset of a wider population to which you want to generalize your results? That is, are you interested specifically in the species in your study, or do these species simply represent a subset of a broader phyla you are interested in? If the latter, the factor should be treated as random.

Note that these are all separate questions from any statistical output, and in fact should be decided a priori.

All of that said, depending on your precise research questions, there are situations in which it is reasonable to "change your mind," and treat a factor as fixed instead of random or to remove it from the model entirely. For example, it may be reasonable to remove the "block" effect in your model if you have strong evidence that it doesn't have a strong relationship with the outcome of interest, in which case it is more parsimonious to remove it.

However, you have to be careful what criterion you use to decide what constitutes this evidence. This is where we get  back to the question of what exactly you mean by "not significant". How did you measure this significance? The most appropriate way to decide whether you can remove the effect from the model is to perform a likelihood ratio test between a model including the random factor and a model excluding the random factor, and test the significance of that test against a Chi square distribution.

Thank you so much Jochen.

Ryan, so interesting explanation. I take your advice.

Ryan, nice post. Only your last sentence seems wrong to me. After all, just as you wrote it, the likelihood ratio test provides a p-value. But the main question is to judge what this p-value tells us about the "significance" (of the tested contribution of the random factor). A judgement based on some unreflected, arbitrary but "commonly used" upper limit for p (like 0.05 what seems a quasi-standard) is not a bit better than simply take a gut feeling and claim a "significance" or "non-significance".

Eventually, the decision about a sensible model is an art, and there is nothing like an "objective criterion" helping there that would be at the same time reasonable or sensible. Looking at tests and p-values usually only distracts us from looking at the important/relevant aspects and from the hard(er) work to find answers to inherently difficult questions. What we need is a careful consideration of the context - not a mindless decision based on p-values.

I agree with you completely, Jochen. I fear I didn't communicate my point very well.

What I was attempting to communicate was merely that, if one wishes to make a judgment based on a p-value, then one should make an effort to ensure that you are using the appropriate test to produce that p-value. That is, the likelihood ratio test is directly evaluating whether the inclusion of that variable "improves" the fit of the model to the data or not, so in this context it seems to be an appropriate test to use (assuming, of course, that the model has a well-defined and monotonic likelihood function, which we can't know for sure without getting more details from Carlos). My main point was that the likelihood ratio test is a preferable measurement to using, say, the p-value provided by the Wald test or equivalent (e.g. the p-value that always appears next to a covariate in the output provided by most statistical software).

I didn't mean to imply that it was the best criterion full stop; as you said, there is no statistical panacea for model building. A well-informed decision would, in reality, be not only examining several different statistical measures of goodness-of-fit (e.g. AIC/BIC, mean square error, the ratio between the variance of the random effects and the residuals, etc.), but also critically examining modeling assumptions in the context of whatever practical, applied non-statistical knowledge is relevant to the research question.

Very good answers in my modest opinion of non-statistician (but metrologist).

In particular, I like Jochem's  statement "the decision about a sensible model is an art" and Ryan's introduction of the possibility to use fixed effects (I would name them the expected values of "systematic effects).

Yes, I agree that statistics come later on the acquired data. First the experience should indicate the way to built a model. I say 'a' measurement model, since it is the prior guess about the experimental situation, something different from the posterior 'data model' - here I am not using prior and posterior necessarily in a Bayesian sense.

In my opinion, it follows naturally an issue that I do not find (at least explicit) in neither replies: I would include in the initial model all the influence factors that sensibly one can detect, omitting none (so, in this case both effects with a mixed model –I like much more mixed effects).

If the data obtained are sufficient, they should provide an indicatation of the degree of importance of each factor. The fact that the model may result 'oversized' should not be harmful for the posterior analysis, unless your experimental design becomes over-complex.

However, missing a reasonably possible influence factor (or randomising it) only for simplicity reasons may undermine the quality of your study.

Finally, you may check the results against multiple models, e.g. but dropping one factor, or by omitting some data (jackknife).

In conclusion, I think that, with a multiplicity of analyses, your data can become more valuable and you can increase confidence in your conclusions/decisions.

You may also need to consider hierarchy. So examine the model with all interaction terms. If a main effect is not-significant (however you determine that), but an interaction term is significant, do you keep the non-significant main effect? I am usually in favor of maintaining hierarchy, but I can envision situations where I might not care much.

Timothy,

being "an art", there is not a universal recipe. If you misunderstand or underevaluate any critical issue in your experiment, you will end with a model more imperfect than you wish. Hierarchy exist almost universally, because almost universally you deal with indirect measurements, so the influence quantities are almost universally interacting according to a hierarchical pattern. One simple case is when, practically always, you have quantities that are responsible for 'corrections' of others.

This, in my opinion, is one reason why I consider dangerous to try to oversimplify the model: put it in there everything, and only the results can help you in understanding what you could consider non-significant in a refinement of the study. I understand that this is a typical attitude of a metrologist who needs to get the maximum level of accuracy, while most people start from the simple, and is not anxious to complicate his/her life.

However, it is much safer the posterior elimination of some factors that proved to be irrelevant, than the omission of some factors for keeping everything as simple as possible. Nature is not simple, the risk is high.

So your usual decision is correct for me.