Dear Md Nur Alam Miah ,

Assumng that your question means "what should be considered in a multiple linear regression model":

There are many issues to consider:

• The assumptions of the model, especially linearity and homoscedasticity (normality is not as important, according to robust exceptions)
• The quality of the predicted model, such as using adjusted R square
• The choice of optimum model balancing complexity against explanatory power
• The best method for establishing the optimal model choice, e.g. backwards
• Whether to include Interaction effects
• Multicollinearity
• Validation of the chosen model, e.g. normality of errors, and whether to use bootstrapping to better estimate the confidence intervals

It is probably best to start with 2 independent variables to gain an understanding of these issues.

I agree with Peter Samuels

You can still look at curves with one regressor/predictor/independent variable. 

You could pick alternative models using subject matter expertise, and compare them using graphical residual analyses.  There you could plot predicted y on the x-axis and estimated residuals on the y-axis.  To avoid overfitting when you look at these graphs, you might save some test data not used for model selection, or to estimate the regression coefficients, and see how well you predict for them. 

Too many predictors often leads to increased variance, and too few often leads to bias.  The best number of predictors really depends upon which ones and their interactions. 

Testing and graphical residual analyses are important, but so is a subject matter justification for the model chosen.  Spurious results are possible, especially with small sample sizes. 

Best wishes.

If a variable to be studied depends upon a single variable then this can be studied by simple regression model.

But if a variable to be studied depends upon more than one variables then this can not be studied by simple regression model. This study is possible only by multiple regression model.

In reality, any variable to be studied usually depends upon many variables though it is is a usual practice to treat the variable to be dependent on a single variable.

Accordingly, multiple regression model is to be considered in order to obtain findings with greater accuracy.

I agree with the opinions of Peter Samuels if the question is

"what should be considered in a multiple linear regression model".

He has beautifully outlined the issues to be considered in the multiple regression model.

In Multiple Regression framework you can not only estimate the dependence of the dependant variable on the input variables , i.e the main effects but also the influence exerted on dependant variable by the interaction among these input variables, the latter information is unavailable from univariate studies pertaining to individual input variables.

The original discussion question concerns the "...multiple regression model. It is more accurate than to the simple regression." But, as I explained above, that is not always true:

"Too many predictors often leads to increased variance, and too few often leads to bias. The best number of predictors really depends upon which ones and their interactions."

In the following, one regressor was often best. The question was when can more than one be justified:

Invited presentation at US EIA, https://www.researchgate.net/publication/319914742_Quasi-Cutoff_Sampling_and_the_Classical_Ratio_Estimator_-_Application_to_Establishment_Surveys_for_Official_Statistics_at_the_US_Energy_Information_Administration_-_Historical_Development

Often more regressors are needed, but they should be the 'right' combination. The best model should be unbiased with the lowest variance, and if it is the best combination of predictors, and there are no data or stratification issues, then one may expect heteroscedasticity to naturally occur. See https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity

I do agree with James R Knaub that multiple regression model is more accurate than the simple regression model since in reality a variable to be studied usually may depend upon many variables (in stead of one variable) among which some may have significant effects upon the dependent variable and some may not have significant effects upon it.

However, if there exists some special situation where a variable to be studied depends upon a single variable then this can be studied by simple regression model.

I did not say that multiple regression is more accurate than simple linear regression. I said that is not always true. It depends. The quote was not from me.

Why should be consider multiple regression model? From one point of view, it is an attempt to fill some research gaps. Of course the outcome is another subject matter.

@ James R Knaub ,

It is clear that multiple regression is accurate when the dependent variable to be studied depends upon more than one independent variable. But, if the dependent variable to be studied depends upon only one independent variable then simple regression is accurate.

However, I have mentioned the reality that in reality a variable to be studied usually may depend upon many variables (in stead of one variable) among which some may have significant effects upon the dependent variable and some may not have significant effects upon it. Accordingly, in reality multiple regression can usually be considered as more practicable to deal with the situation. If multiple regression model is applied to a situation where the dependent variable to be studied depends upon only one independent variable then the coefficients associated to the other independent variables will automatically either be 0 (zero) or negligible (insignificant).

I am also of the same view with that of Aristidis Matsoukis that to consider/use multiple regression, from one point of view, is an attempt to fill some research gaps.

I like to add that in reality multiple regression can usually be considered as more practicable to deal with the situation. I have explained it in my earlier answer.

Above I included a link to an invited presentation to an agency that does official energy statistics for the US. (I had worked there and was asked to give a seminar for the mathematical statisticians.) In many cases, one regressor is great because it is the same data item as in the frequently occurring sample, found in a less frequent census. Heteroscedasticity is important, and no intercept term. It is a very simple but effective model for a production environment for a great many small populations which differ in many ways. (By the way, a form of small area estimation is used to borrow strength as noted in the presentation and references. However, there are many different populations for which this is problematic.)

So this is an example where one regressor is usually best, and it covers a huge part of the US energy industry, reporting official statistics for finite populations. Tables with perhaps more than 1000 aggregate values are published every month. Much is published weekly.

So, I'm just saying that the statement in the original question, that says a multiple regression is more accurate than a simple linear regression, is not always true.  For example, sometimes there is only one regressor, and it is a very good one if it is the same data item in a previous census, used for a current sample. 

Further, as i think i already stated, in machine/statistical learning it is warned that adding independent variables often increases variance. 

In Brewer(2002), Combined survey sampling inference: Weighing Basu's elephants, Arnold: London and Oxford University Press, on page 110 he (Waksberg Award winner, Ken Brewer) argues against using an intercept term when not needed, and the range of x is limited, because it can cause a loss of efficiency (greater variance) by letting the intercept "stray."  He also says, on pages 109-110, that "It is well known that regressor variables, when introduced for reasons other than that they have appreciable explanatory power, tend to increase rather than decrease the estimates of variance." 

(I'm assuming that under "Fair Use" I can quote that much of a copyrighted book here.)

So, just as in machine learning, in Brewer's book for survey sampling, there is a warning about adding independent variables when not needed. 

Generally, there is a tradeoff between bias and variance when considering adding or removing independent variables.  That tradeoff is a big topic in statistical learning. 

The summary

" The statement in the original question, that says a multiple regression is more accurate than a simple linear regression, is not always true. "

in the explanation provided by James R Knaub

is acceptable.

However, I have not yet known that in statistical learning adding independent variables often increases variance.

Is there any logical explanation?

If yes, the explanation is requested.

Dhritikesh Chakrabarty - In answer to your request:

OK, I went to a paper I wrote where I used some statistical learning references, and the like, for part of it, and I put together these references:

Fortmann-Roe, S.(2012), “Understanding the Bias-Variance Tradeoff,”

http://scott.fortmann-roe.com/docs/BiasVariance.html

Shmueli, G.,(2010), “To Explain or to Predict?,” Statistical Science, Vol.

25, No. 3, pp. 289-310. Institute of Mathematical Statistics, DOI:

10.1214/10-STS330,

https://www.researchgate.net/publication/48178170_To_Explain_or_to_Predi

ct, 23 pages

Wood, F.(2011), “"Regression Estimation - Least Squares and Maximum Likelihood,” online slides, University of Oxford,

http://www.robots.ox.ac.uk/~fwood/teaching/W4315_Fall2011/Lectures/lecture_3/lecture_3.pdf

 Hastie, T., Tibshirani, R. and Friedman, J. H. (2009), The Elements of

Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed.

Springer, New York.

In the appendix for Shmueli(2010), the left column of page 20 (as marked in this version she was apparently allowed to upload), she gives one simple example that shows one such case.  It is for the case of two regressors being correct, rather than one, but it is the only simple example that shows bias-variance tradeoff in an application that I know.  Other references above showing how tradeoffs occur overall probably won't address your question as well.  But you may want to look at Hastie, Tibshirani,and Friedman, and you can go to the link above for Fortmann-Roe,  which is good, but mostly is not on regression models.  As I recall, I think the Wood graphics in the first part were good. .

But I think the appendix in Shmueli will quickly address your question. There is a bias-variance tradeoff, and sometimes you do better overall if you reduce complexity, and sometimes you do better if you increase it. Also recall the Brewer reference on survey sampling. On pages 107-110 he also notes that you need good reason to add an independent variable. I suppose that was supported by his extensive experience.

Anyway, you can look up bias-variance tradeoff other places as well. Increasing the number of independent variables tends to increase variance. If you have an unbiased model, that sounds like trouble. The Shmueli example was for a biased one-predictor model. You would not do that. So if you have an unbiased one-predictor regression, you better have a very good reason to add another predictor.

Cheers - Jim

To make it more intuitive, I think the Fortmann-Roe paper said it is about "underfitting and overfitting." I suppose if you put in too many predictors you can be more likely to avoid "omitted variable bias" by overfitting, thus having estimated (random factors of) residuals average well, to zero. but you can vary further with larger residuals due to the greater possibilities with more predictors. - Just thought I'd give the intuitive aspect a try, but it is a bit vague.

@ James R Knaub ,

There is a significant information in the answer provided by James R Knaub.

Neither the Fortmann-Roe paper nor any link of Fortmann-Roe paper, which describes about underfitting and overfitting, is not known to me. Any type of information that can enable me to read the paper is requested so that the concept of underfitting and overfitting can be understood.

Thanks to everyone who gave their opinion.

Can we use this model for case study base analysis in a region?

this can be helpful https://www.reneshbedre.com/blog/multiple-linear-regression.html

Thank you sir

Fran, et.al. -

When I mentioned above that if a simple model was unbiased you might not want to add predictors, I meant that if the expected estimated residual is zero, based on the sample used (but the sample used can matter), then you don't want to add complexity/more predictors unless you have left out an important predictor. But that was silly of me because you can always easily obtain apparent expected sum of estimated residuals zero, or under specific conditions of heteroscedasticity and intercept, exactly zero, for any specific sample, even for more complex regression cases. So it isn't so apparent that there isn't some bias from the 'true' model.

However, additional predictors tend to add to variance, unless they are really important, which Brewer, referenced above, notes, and I have found to be true, and is part of the "bias-variance tradeoff" often noted, if you remember that an "important" variable can drastically change this.

You also don't want to overfit to the particular sample you have. You'd want good reason to believe your model fits the data, say by looking at a "graphical residual analysis" scatterplot. But you want to avoid overfit to your particular sample, and "cross-validation" is helpful there. An internet search on "graphical residual analysis" and on "cross-validation" should provide a great deal of good information.

Cheers - Jim

As for me, I usually identify the main predictors from the theoretical frameworks or theories. Also, I may experiment, add or remove to observe R-squared values or similar identifiers of the goodness of fit. Moreover, logically, one phenomenon usually can be explained by multiple independent variables that stimulate using multiple regression.

I also subscribe to the views of Ibrahim Niftiyev. Infact, once your major interest is joint effect and relative effect, multiple regression analysis is adequate

The recursive least squares algorithm (RLS) allows for (real-time) dynamical application of least squares (LS) regression to a time series of time-stamped continuously acquired data points. As with LS, there may be several correlation equations with the corresponding set of dependent (observed) variables. RLS is the recursive application of LS, so that each new data point is taken in account to modify (correct) a previous estimate of the parameters from some linear (or linearized) correlation thought to model the observed system. For RLS with forgetting factor (RLS-FF), acquired data is weighted according to its age, with increased weight given to the most recent data. This is often convenient for adaptive control and/or real-time optimization purposes.

Application example ― With a single correlation equation ― While investigating adaptive control and energetic optimization of aerobic fermenters, I have applied the RLS-FF algorithm to estimate the parameters from the KLa correlation, used to predict the O2 gas-liquid mass-transfer, while giving increased weight to most recent data. Estimates were improved by imposing sinusoidal disturbance to air flow and agitation speed (manipulated variables). The proposed (adaptive) control algorithm compared favourably with PID. Simulations assessed the effect of numerically generated white Gaussian noise (2-sigma truncated) and of first order delay. This investigation was reported at (MSc Thesis):

Thesis Controlo do Oxigénio Dissolvido em Fermentadores para Minimi...