Hi Alex,

It is not clear to me from your post whether you have actual data or are you trying to generate data for simulation? Please clarify what you are trying to do with as much specificity as possible.

Ariel

Hi Ariel, data is empirical, I have updated the question to provide more specific details of my problem. Thanks

Hi Alex,

You make the statement "I am interested on determining the relevant variables to explain the process. If these variables are truly accurate surrogates of the process they should be robust regardless of the environment. Thus I was thinking on a model where the environment does not apply as a predictor variable."

The covariates in your model are not surrogates for the dependent variable ("process"), but explanatory variables in the sense that they explain the variability in the dependent variable.

If I understand you correctly, you are looking for the best combination of covariates to explain the dependent variable, correct? If so, then I would suggest using a LASSO or Least Angle Regression algorithm. This will find the best-fitting model, and provide several indices of fit (such as AIC, or BIC or R2, etc.). You could also consider boosted regression, but that is somewhat of a black-box.

Ariel

Yes, that is part of it.

But my true problem is orthognality (the lack of...) in environments.

My main trouble was to take into account the unbalanced design without explicitly integrating the "environment" variable as a predictor of the model. Besides, if I use as error term, I am reducing ly chances of finding significant patterns, because replicates are not repetitions, but independent plots, so I think this is not desirable.

I have carried a model like the following:

model_3 = glm(Process ~ Var1 + Var2 +Var3 +Var4, + (Var1 + Var2 +Var3 +Var4):environment, family = binomial)

While this is somehow satisfactory considering what variables are significantly related to the predicted process in comparison to graphical exploration of data, I am still hesitating whether this is correct or not.

Any feedback on this?

I am not an R user, so I can't comment on the command you are using. Have you considered a multi-level model where you use random effects for the environment?

Yes, this was one of the procedures. Nesting replicates within environments, and build a generalized mixed model, where random effects were provided by the replicates. However, considering environment replicates as "random" is somehow misleading as replicates are independent locations, and variability is relatively large, and homocedascity is not met.

"I am interested on determining the relevant variables to explain the process."

This is the cruicial sentence. There are two words that need some elaboration: relevant and explain.

What do you mean by "explain", precisely? It could be the reduction in residual deviance(variance) or it could have some substantial meaning (related to the science, to functional models).

And you need to specify what you mean by "relevant": what amount of deviance-reduction do you consider relevant? Or what size of the estimated parameter do you consider relevant? Or is relevance related to the meaning of the predictor in some functional way?

So if you look for substantial answers then you need substantial arguments (not statistical arguments). Looking from the side of statistics, there is no single answer, nothing like "one size fits it all". Different models may have different pros and cons, one model may be better for one purpose, another model for another purpose. So you will need to specify the purpose of the model.

Here it seems that you wish to get a model as simple as possible. The auxiliary condition surely is that the "fit is good enough", what I like to translate as: "the prediction based on the (remaining) predictors should be precise (and accurate) enough". You can analyze the confidence intervals or prediction intervals for the predictions for different models and see if this is "good enough" for whatever purpose you have in mind. Then you can chose the simplest model that gives such "good enough" predictions.

You have "only" 4 predictors and you are not looking for any interactions, so there is a rather small numer of possible different models (16 I think). You can actually fit all these 16 models and see which one is best serving your demands.

Regarding the environment: I think that including this as random factor is useful if there is some correlation of the response with the environment.

I guess you are right, and I stayed too « vague » on the definition of the problem.

The predictor variables are indices derived from inherent properties of the compared elements.

By relevant and explain I meant that I wanted to understand whether these indices have a causal relationship with the explained variable, and that eventually, I’d be interested in understanding which set of predictors is likely more accurate to make inferences of the explained variable. So among the cases mentioned I am in the third: relevance is related to the meaning of the predictor due to a functional hypothesis.

The predictor variables are expected to relate to the explained variable in no matter what conditions. Thus, it should be irrelevant whether I add environment or not in the models. However, I know, that this is not true. The variables interact with the environment, and therefore, their quality as predictors depends on the context.

This provides slightly different outputs depending on how I build the models.

Considering environment a random factor resulted, I’m afraid, in an increased type II error risk. I say this after comparison with models without random effects, or event adding environment as a predictor variables, as well as graphical exploration of the data.

Yet, adding environment as a variable, resulted in a consistent effect of environment as a relevant variable, and the reduction in the signification of other variables, or even some changes.

This suggests that important variations occur between observations, but interactions take place with environment.

While this is interesting according to my question, my dilemma still is, that I have different number of observations for each treatment, but using environment as random factor is too conservative. I was thinking on adding a weight variable or similar to observations, but I am uncertain whether I can do this for a variable not present in the model.

"Process = success/fail variable" that is your outcome variable, correct? And you are trying to see if the environment changes the relevant predictors of this variable, am I understanding this right? Well, create a variable called "Environment": 1, 2, 3, 4 and include this variable in a logistic regression analysis. In such model you can even include interaction terms between the Environment variable and any other predictors of interest. I don't see how ANOVA can work with a dichotomous outcome variable.

generalized linear model using number of success/fail

Iteractions of environment with all predictors use more df than ly data set can afford.So I cannot test all possible interactions, unless each predictor is tested individually.

No you cannot test ALL interactions, that is why you have to look at your theory and see which interactions are relevant and which ones are not and test just the ones that are important from your research question point of view. If you have too many variables, do factor analysis and reduce the number of variables.