I think you should give some more information like the number of variables, number of observations, type of your predictors, characteristics of residuals etc.

Also what do you mean by meaningful and opposite meaning? Why not give number of variables/ names of variables(like x1,x2 ...)  included at each model?

I suppose you are talking about a linear regression model.

The most important criterion you have is "The ones that are significant are meaningful."

You need to clarity what is your study purpose. For testing specific conceptual model or hypothesis, all you need to do is select a a statistic model matched the concept. For exploring knowledge, we should keep balance between fitness and degree of freedom. For prediction, you need to select models depended on targeted population; if that is just same as your data source, cross-validation error value is very informative.    

Amir,

There are several methods for selecting variables for your model generally, they can be divided into:

- Testing-based procedures, and

- Criterion-based procedures.

I believe the criterion based procedures are more practical than the testing-based procedures. It may be useful to provide more information regarding your models and variables and reporting different model selection criterions in order to select the adequate model. Hope this helps!

It is necessary to know the sample sizes of your data. If all the required conditions are the same, the analysis with the least  standard error should be the most reliable irrespective of the  R squared values. Significance level issue is, to me more important. Y

I am not entirely sure what you are doing, but it sounds like you have some data and you are running against all the tests that you think are important or might tell you something. I call this "looking for the result I like." The test or tests you use should be determined before you run experiment, this is part of the experimental design. The type of data or more correctly the type of distribution the data exhibits will define which tests are appropriate. If more than one test can be used, you decide this is the test I am going to any why. What you shouldn't be doing hunting for the answer you would like to see.

Thank you all.

I have 11 variables and 132 observations, I know the number of observations are at lower side of spectrum but I cannot have more observations and I reduced the number of variables through some statistical analysis and prior knowledge, from 14 to 11. I added the regression equation in the updated file, attached. I used 9 different techniques including multiple regression, robust regressions (4 different types), shrinkage method, and different weighted methods in order to estimate the coefficients and regression model without considering the two-way interaction yet. In fact, the value of launching a new vehicle coefficient is positive, based on the data and different statistical analysis. It means that the dependent variable increases while launching a new vehicle. This fact can be seen only when I'm using methods 1 and 7. Therefore, I decided to use method 7 because it has a better statistical model performance comparing to 1, multiple linear regression including different transformations. Thank you.

Some see 'data science' and 'statistics' as different areas and, given the distinction, you may have crossed the line into data science. 

There are pitfalls in playing with the data too much before coming to a conclusion.   Unfortunately the best data science is tormented with the problem of over fitting. If you try a data set enough ways you will eventually get a model that only fits that sample. In the case of your attachment, it is difficult to say the degree to which that was possible, but your peers are going to hesitate to accept your conclusion. They might suspect that it could be over fitted and might have no generalizable meaning. On the other hand it is valid data science if you treat it as a 'training set' (analogous to an hypothesis) and move on with more (new) data to confirm the model you have developed. 

I agree with Lewis. I cannot count the number of times I have had someone come to me with some data and ask, "what does data tell you." *sigh* It tells be you should talked to me before going out and spending the time and money gathering data. The best use of this type of data is discovery research. Often you do not know the type of distribution that data will exhibit, you only know what type it will typically exhibits. This is one the reasons for making histogram of the data before you get to far into the experimental process. The absence of this formality is part of what I will be investigating in my dissertation.

I also agree with Lewis; however, I believe there could be some overlapping between theses two, 'data science' and 'statistics'. For instance, In my specific case through data visualization and comparing the mean through t-test, it was obvious that Japanese plants had a better performance compared to American. However, at the beginning, I was not able to draw this conclusion from the statistical models because of suffering from a low number of observations for Japanese plants, 22 versus 110. Hence, I decided to reduce the number of variables (from 14 to 11) through PCA and quantile regressions which the results were fruitful. However, I'm in the impression of using a regularized technique with a combination of robust regression and performing cross-validation solve the overfitting issue.

Moreover, in the practical world, you may not be able to do a nice DOE and lay out your design in advance. You were provided with a sample dataset, from a population, and want to explore some hypothesis and sometimes come up with an estimation model.

How much time do you have? You have started a good analysis. To finish you have to figure out which model is giving the correct answer. I would start by examining the model assumptions and testing them. Can you eliminate some models as inappropriate because it requires some feature that is lacking in your data. Just because you can plug numbers into the stats program does not guarantee that the numbers barfed back at you have any meaning.

If you are lucky, you are down to 4 or 5 models. Now you have to figure out why they are trying to tell you different things. I would plot the observed and predicted values for each model. Maybe there is a single data point that is highly influential? Maybe there are local minima/maxima and some models are more likely to get trapped. You can play with each model and rerun the analysis after first removing one or a few data points. You might try using each model in a simulation and see how the models differ. You could try using one model to simulate data and use the other models to analyze the simulated data.

You should be able to eliminate a few of the options by this point, but you might still have 2 or 3 models. You could present all of them and discuss what the differences might mean. Alternatively you could use your expertise in this field of study and select what you think is the most reasonable model. If you feel exceptionally honest, you could mention the fact that there were several competing models and you chose what you thought was the best one. However, this risk reviewers wanting to see everything. A compromise would be to focus the manuscript on the model that you think is the best, but include other options in supplemental material.

One has to be careful when fitting data to multiple models because you may get the best fit for your data which unless quite large will be biased. One way around this is to select a model and then test it against new data. In other words, the best fit to your data may not be the best fit to the population.

@John, Agreed. However, running only one model doesn't change this. The only difference is that you do not know if there are other possible answers. That said, the temptation in running multiple models is to choose the one that you like the best. That would be a very bad choice.

There is always the assumption that your sample is a good model of the population. This is not guaranteed, but the possible difference between the distribution of your sample and the distribution of the population declines with increasing sample size.

You have had a lot of good advice on this post from other colleagues. I will add my little advice as well - in my opinion no model is ever perfect and certain aspects of your data can be represented well by the model, while others will not. As a rule of thumb for non-hierarchical models if number of covariates go over log(n) - where n is the number of data points - you will start to overfit - this however is a rule of thumb only. We have a few posts about this on our blogs at https://laplacebayes.wordpress.com/category/model-checking/ and https://datacatz.wordpress.com/tag/model-comparison/ - you may find it useful.

Good luck

Thank you Umar. I'm thinking it's Ln(n), not log(n), am I right? Can you share a reference for that? Thank you again.

Yes natural log sorry I just wrote the R function :) it should be in the chapter on model checking, I think 7 or 8 in Bayesian data analysis Gelman 2013. It also depends on how many parameters are being estimated in a model, which is also a function of the data. If you read that chapter it explains it in more detail.

A Possibly Relevant Aside

I suspect you are aware of Freedmans Paradox (1980 ), and it does not seem to apply here. However, being as it relates to the flexibility (and lack of flexibility) of regression techniques. it is tempting for someone following your posts to fall into this weird pit:

http://www.public.asu.edu/~gasweete/crj604/readings/1983-Freedman%20(Screening%20Regression%20Equations).pdf

(The above is one URL, so cut and paste getting rid of any line feed.) also see he more modern verification using more data: 

https://www.r-bloggers.com/freedmans-paradox/