I prefer using backward selection because all covariates are entered then the model is reduced. 

You can use something called "best subsets". This will create many different models which should have several different terms that are similar.

The regression models you produce are already stochastic. If you didn't collect data well, or didn't enter appropriate factors into the model, your model could be less than desirable. For instance, I had an experiment where every Wednesday the results were significantly different from other days. That happened to be a day when we had 2-3 sources of contamination regularly released. 

What is your goal for the experiment? If it is finding optimal settings for your process, it is important to remove insignificant terms. If your goal is prediction, removing insignificant terms is not that important.  

If you have a good designed experiment, ie. you used stats software like Design Expert or JMP, your factors will not have any covariance issues among different terms, main effects nor interactions.  

In my opinion, using "Best Subset" is a "fishing expedition".  One should have some sort of theoretical or biological hypothesis before any analysis.  Modeling just verifies or refutes this hypothesis.  I use backward elimination in SAS because of the info it gives when choosing a variable to be removed.  One can gain much info this way.  We all must remember that Statistical Modeling is is not  meant to give definitive answers, it just points to a possible answer.

If your data is orthogonal or nearly orthogonal, then best subsets and backward elimination will give the same "best" results. If the data are not orthogonal, then you have major issues with backward elimination and best subsets. However, best subsets will show you that there are several significantly different "best" models. Backward elimination won't.

I've been in discussions with SAS statistical consultants, among others. They tend to use best subsets for the reasons I described above. It almost acts like a Monte Carlo type simulation too. If 20 models say factor A is significant and 80 models say it's not, would you use factor A in our model? 

If all you use is theory to create your models, then there is an assumption that you already know what is going on, which we don't. If you look at economics, by using theory to guide our model making, we can look at the same data, use 2 different theories and come to 2 different conclusions.  Both models give biased results, neither will capture the true relationship behind the data. Using either model will lead you down a path. You may get closer to the "real" truth or further away. But, you'll never know for sure because you are chasing after "perceived" truth. 

There are several stochastic design of experiments algorithms based on Kriging.

Since Kriging deals only with Gaussian process, the theoretical calculations related to the estimation error can be done very easily.

Depending of your final goal, you can choose several algorithms. For instance:

 - Expected Improvements (EI) ([Mockus 78, Schonlau et al. 96, Jones et al. 98])

if your final goal is an optimisation

 - Informational Approach to Global Optimization (IAGO) ([Villemonteix, Vazquez, Walter 07-09])

if your goal is sample where the largest uncertainty reduction is expected.

These algorithms do not select directly influential variables but they tell you what is the most informative next experiment. The non influential variables will not increase the number of needed experiments.

I do not know if there are Kriging algorithms which directly select the influential variables. But I think it should be possible since Kriging enable to calculate what is the gain of information when you do a new sample.

If you use Matlab or Octave, you can use the toolbox STK :

http://sourceforge.net/projects/kriging

http://sourceforge.net/projects/kriging

Hi Watheq,

You might want to take a look at machine learning approaches.  For example, there is a procedure called Regression Trees.  Regression trees can be taken a step further to something called Random Forests.  Their result is stochastic in the sense I think you mean.  Their main advantage is that they a very robust and don't require many of the assumptions that multivariate linear regression requires.  

There is also a procedure called Multivariate Adaptive Regression Splines (MARS) that was developed in the chemometric discipline.  Lastly, there is Chi Square Automatic Interaction Detection (CHAID).  These two are also quite robust.  There are commercial and open source programs to implement the procedures.

Good Luck

George

Thank you all.

Actually, I came up with a Bayesian procedure for stochastic experimental design through adopting Bayesian Model Averaging (BMA).

By BMA, it will be easy to generate multiple models by sampling from each independent variables to get different coefficients sets.

The best model has the maximum posterior probability and minimum BIC.

I applied to a physical process (Enhanced Oil Recovery) and it works much better than multiple linear regression.

Thanks again.

Watheq.

BMA sounds like it is derived from resampling procedures developed by Bradley Efron and others.  For BMA, will you be using WinBugs?

George

George,

No. I did it in R-statistics through BMA package.

There is another methods that considers MCMC just for eliminate the non-influential predictors without estimation the predictors coefficients.

Best,

Watheq

http://bms.zeugner.eu/

Thanks a lot for all your responses, I appreciate your help.