What do you mean by « ranking » these variables? The larger effect, the better? The more significant, the better? Other considerations?

Yes, larger effect, the better. More significant, the better.

That's not the same thing. You can have large but not-significant effect, and vice-versa. You must choose, or find a way to combine these two criteria ina single one.

Ranking on p-values is quite easy: just test all variables one by one and rank the results.

Ranking on effects is more difficult because you have to consider the effect of the different scales of your explanatory variables.

In both cas, interest of such a ranking is difficult to evaluate. You should better think more carefully about what is the final aim. What is this ranking for, for instance?

Are your independent variables correlated? The regression model assumes that the independent variables are independent of each other.

I would recommend looking at some references on variable selection. You might start with the attached paper and the book in the link below. Best wishes, David

http://www-bcf.usc.edu/~gareth/ISL/

There are four basic approaches to model building. 1) Theory: you know the model, the data must fit. 2) Forwards selection: you add the most "significant" variable first, then the next, and so forth until you run out of variables or no other variables are "significant." 3) Backwards: Start with the full model and remove terms. 4) Stepwise: add variables, then check the model to delete variables, before going back to the pool of available variables to look for the next "significant" variable. I have used parentheses about significant because there is no limit on the criteria for entering or removing variables. It might be p-value, or maximizing R2, minimizing AIC, or some other criteria of your choice. You might be able to accomplish this task using something like machine learning, but this approach is not common in agriculture related insect science.

You have built the model, but that might not answer your question. Your dependent variable might be insensitive to variable X1 and therefore X1 must change a great deal to cause any effecct in the dependent variable. However, the dependent variable is very sensitive to any change in X2. Both variables may have a p-value of 0.0000004, and be uncorrelated with each other. Which one would you consider influential? That might depend on how difficult it is to change X1 and X2. You might be able to answer this question through simulation.

All of this is fantastic, except for several problems.

1) I expect that your independent variables are correlated. So, if X4 and X17 are correlated with an r2=0.73, which one is important/relevant and how do you know which one (or possibly neither) is the cause-effect relationship with the dependent variable?

2) It is easy to overfit your model. The number of variables in your model should not exceed the number of data points. In general, the number of variables in your model should be a small fraction of the number of data points.

3) With a small sample size your estimates of mean and standard deviaiton are uninformative about the underlying distribution. All statements will begin with "assuming that X3 is ________ distributed ...."

4) A small number of replicates does a very bad job of estimating the probability of an unusual occurrance. It is unlikely that my sample size of 10 will detect the 1 in 200 probability that a patient will die due to this new drug that I am testing.

@Timothy I'm afraid your model building information is somewhat out of date. Suggest you see the following references. Best wishes, David

http://www-bcf.usc.edu/~gareth/ISL/

@David, perhaps, though I did include machine learning at the end of the first paragraph, and I freely admit that the core of suggestions were somewhat old. Thank you for the references, though it will take a while to go through the stats book.

I have a manuscript that used a non-linear regression method first published in 1987. The manuscript has been stuck in internal review because it is "too complicated." Mostly what is desired is a bar chart with treatment means and a t-test for all pairwise significant differences. In some cases, p-values are relaxed to 0.1 or 0.15 for experiments with 4 replicates per treatment. Fast, simple, and able to convey "complex" ideas in 5 minutes or less. People don't want to hear that their entire experiment is little more than a random number generator. This may or may not make any difference to Saad's case. However, if the research will be reviewed internally or externally, the method should not depart too far from established approaches. Alternatively, publish a methods paper first, and then publish the paper that used those methods. I suppose one could also do a review where the old method is used and then compared to the result of reanalyzed data. Something that bridges the knowledge gap of potential readers/reviewers.

Here are two simple questions for David: Given that Saad has 4 replicates per treatment, which method (OLS, Bayesian, or machine learning) will fare best under this small sample size? Would your answer change if you knew the distribution of each variable beforehand?

My closest experience is with computer intensive methods like bootstrap or randomization tests. These methods are very poor choices with small sample sizes no matter which underlying distribution is assumed.

@Timothy. I am very sympathetic to your comment "Fast, simple and...". I have a paper out now that nobody will review because the Stats appears too complicated. I have a second paper out that was criticized because I didn't use the most advanced method proposed. One does the best one can. i usually try to give the best answer I can to the question I am considering and then use reviewers comments as best I can to give the final "best" answer. The Simplest is not always appropriate however. Saad's question is complicated and I think should be divided into two parts: 1. Are there enough observations to fit the model? and 2. What variables are most important? Considering 1, a reasonable rule of thumb is to have 5 obs per IV to get a reasonable fit with an OLS regression (I have seen suggestions running from 2.5 to 15, 5 seems to work for me). Spse he has 10 IVs then I would feel that 50 observations is about the min needed to fit the model using OLS. The number of replicates only matters here in that more replicates mean more observations. Eg see Mendenhall, Intro to Linear Models and the Design and Analysis of Experiments, Duxbury. Notice here that the number of replicates is not sufficient information to answer question 1 assuming, say, a random block design. We need to know the no. of Treatments, Blocks, and Replicates per block,treatment combination to compute the no. of observations total in the X matrix. see Mendenhall, p216 Excer. 23a. Now consider question 2. Here as was pointed out above, What you mean by important is critical. I would refer the interested reader to Gareth James, etal, Intro to Statistical Learning cited in my earlier response for how methods like lasso, gradient boosting etc can be compared. See my cite CanGen1 above for an example of why stepwise is a poor choice of method that is due to Austin and Tu. I hope this helps a little. Finally James et al discusses in detail the methods Timothy asks about so I refer the reader to the experts which James et al really are. Best wishes, David PS I almost forgot. Under OLS all IVs are constant. Assuming the IVs have error distributions puts you in Errors in Variables models. For a discussion, see the link https://www.google.com/search?q=errors+in+variables+models&ie=utf-8&oe=utf-8&client=firefox-b-1

In answer to the question which fares the best, I would be tempted to say gradient boosting but that is also subject to other things. Please see the references.

We must also keep in mind that Saad still did'nt explained why he wants to rank the variables and what he wants to do with this ranking. Answers based on variable selection are probably what he wants, since it seems the most natural idea for most of us, but who knows if he really wants to select variables, and not just simply rank them for another reason?

Stepwise, LASSO and similar methods will not, as far as I know, rank variables. They will just select a subset of them, hopefully a meaningfull one.

@David: a question about errors in variables

The link did not work for me, so I found a Wikipedia page.

Why is measurement error any different from other unmeasured sources of variability? It is all a Simpson's paradox sort of thing where the unmeasured variables can possibly result in misleading relationships in the observed data.

1) It makes a difference if Bob or Sally ran the equipment.

2) It makes a difference if Amanda used the HP5890 or the Shimadzu Gas Chromatograph.

3) It makes a difference if .... and the list goes on from the brand of pipette, to the leaf area meter and how it was callibrated and how often it was recalibrated, shape of the greenhouse, orientation of different fields, soil type, and so on. Even if it was all done by machine (autosampler on a GC, GPS on a tractor) there will be some additional variability introduced at every step of conducting an experiment, and one should expect that the variability changes over time as parts wear or get dirty. None of this is directly associated with the primary research question to relate nitrogen fertilizer input to yield output.

What I want is yield=nitrogen

What I have is yield = nitrogen + person + equipment + year + ...

My data is good for the first question, but essentially I have a single replicate for most of the variables in my model.

Essentially you have the real effect plus how that effect interacts with your experimental design that interacts with your methods that interacts with your analysis of the data to yield a result that you interpret. Experimental design includes things like sample size, and how samples were chosen (random, arbitrary, dispersed, purposive).

Did I miss something that distinguishes "measurement error" from "other sources of unmeasured variability?"

@Timothy: I think the usage for "measurement error"/"error in variables" is that it changes slightly the model writing, compared to your approach.

Assume a straight line, Y= a X + b.

Usual regression says that, since X is "perfectly known", you can write p(Y/X=x) = a x + b + epsilon and play with that, until, for instance getting to OLS with usual assumptions on epsilon.

However, if X has uncertainties on it also, even if the identity above still holds, we don't know which x value to use... We would like to write something like p(Y/X in[a,b]) = ..., for instance.

Of course, one may say that x = Xobserved - epsilon_X, so it would lead to p(Y/X=x) = a Xobserved + b + a * epsilon_X + epsilon = a Xobserved + b + epsilon', but the epsilon does not have the usual OLS properties (for instance, because of the a*epsilon_X term, they are not homoskedastic anymore).

That's not exactly the same thing than adding additional variables… Unlesse you could write a special column for the epsilon_X values, of course (but in that case, you would'nt separate it from Xobs, instead compute x directly, right)?

However, for the moment, I didn't have the opportunity to use such models, so I may be wrong on their basis and interpretation.

I had always thought of epsilon as error from all sources. It is everyhting from biological noise to Bob recording the number 123.7 as 132.7. With luck the latter issue is caught, but no guarantees. If one is very careful and has a large budget, then one could start adding variables to break apart epsilon into new variables that explain more the the error term. The problem with heteroskedasticity is not unique to measurement error. Most treatments don't just change the mean, they also change the variance. It is just that this is hard to measure when you have 5 or 6 replicates.

Somewhere I forgot the assumption that epsilon was independent of X, but in most cases this is not true. Error always consists of the effect of a large number of unmeasured and uncontrolled variables. One part of the problem is that people are accustomed to think at a scale roughly proportional to their body size. It is very hard to relate to an insect that is 1.1 mm long, 0.5 mm wide and weighs 0.0005 grams. I might measure the temperature outside, but this insect lives in the boundary layer of a leaf/stem and that modifies the insect's environment. It is more humid, not as hot nor as cold as the extremes measured by my thermometer. I can put my insect in a growth chamber. While the growth chamber looks uniform to me, I can still detect differences in plant growth in different parts of the chamber. This will translate into differences in the insects feeding on that plant. I am basically stuck, and at some level I have to admit that there is the possibility that none of my research results are correct, but everyone else is also in the same boat. Within this context I still don't see how measurement error is uniquely different. How does the error from something like "using a weather station to estimate the environmental conditions of an insect living in the boundary layer of a leaf that is 1.2 meters distant from the weather station" differ from something like "the effect of the (unmeasured) soil water saturation on the response of a plant to additional nitrogen?

@Timothy Errors in Variables means errors in the X's not just Y as in OLS. Best, David PS This suggests people use OLS sometimes when errors in variables models are more appropriate.

@Emmanuel. All variable selection methods of which i am aware select their variables in some order. See the references in my first post. Best, David

@David, What I start thinking is that the OLS model is never appropriate because there is always error in X. The scientist is never able to apply 60.0000000000 kg per hectare. The nurse is never able to inject the patient with 3.50000000000 cc and that pill is never exactly 15 mg down to the 10^23 decimal place. There is always some error. If one does research with 50 replicates/treatment the effect of a treatment-to-treatment variability in pill size of 0.002 mg is too small to detect, but the variability is still present. Whenever something is applied/done/measured there is always error.

@Timothy, Sorry I was in hospital and just now saw your comment. My reply would be to quote George Box. "All models are wrong but some are useful". Best wishes, David