Neelabha -

When considering outliers, one must be careful not to throw out data simply because it is inconvenient. When looking at totals, one has to at least impute for each case. But if you are only using data points to determine a regression 'formula,' then it may be more acceptable to drop a few. If, for example, you know that one or more independent variables were collected in error, then the predicted-y_i value is unreliable, and thus it cannot be compared to y_i, and all must be dropped for that case _i_. If you have a reliable way to impute for each independent variable value in such a case, then more uncertainty is still present, though a colleague and friend of mine once dealt with quantifying that in a simple case, when totals were needed, but it gets messy, and with multiple independent variable cases for the same data point _i_, it would seem that reliability would be very problematic, especially when first selecting a model. If the dependent variable is a potential outlier, then obtaining ei = yi - yi* is again a problem, so you may have to drop it. My point is that yi and predicted-yi are needed in a graphical residual analysis to see what regression your sample is suggesting, and cross-validation after that can be used to try to avoid overfitting to a particular sample, but you have to be using reliable data.

Perhaps you could look at some under-specified models, where you may have substantial omitted variable bias, just to be able to have larger sample sizes, and see if that provides more insight for the full model. Or, if a very simple model is adequate, you don't want variables just thrown in anyway, as that tends to increase variance.

These are just some impromptu suggestions.

Best wishes - Jim

The first attempt would be finding an interpretation or a reason for outlying. Could you find the rationale behind them? If so, consider this reason for subsequent analyses. If not, continue to conduct exploratory analysis. How many are the outliers? If relatively very little (ratio to the total observations), removing is the easiest solution. However, both analyses (without and with outliers) are generally recommended.

BTW, using a robust regression, say least absolute deviation, may be helpful, but it just mitigates the damage. If you are trying to do model selection, you might be able to do better. But robust regression could still be a consideration.

Also, beware of points with high leverage and/or in the case of weighted least squares (WLS) regression, large regression weights. They need to be as reliable as possible considering their potential influence. (Note that heteroscedasticity, causing use of WLS regression, is natural and expected because the predicted-y values are size measures which are responsible for the essential heteroscedasticity. See https://www.researchgate.net/project/OLS-Regression-Should-Not-Be-a-Default-for-WLS-Regression, and the various updates there.)

James R Knaub Babak Jamshidi

Thanks for your suggestions.

Very simple: if the measurements are OK, leave them in; if you have a good reason to believe the measurements were wrong, repeat them if possible, otherwise throw them out.