Consider the mathematical regression model: Y=a+bX+e , sometimes, the data exposed to outliers or violation. Can look at the above model as two parts; i.e.; Y=YL+YE, the first is linear regression representation YL=a+bX as an explained information effect, whereas the second is the error representation that we with to make it more homogenous and gain more explanation for the remaining information YE= m(e) ; (say; nonparametric smoothing function). Now, can we consider this approach as a new hybrid regression that extracts all potential information and then absorb them consequently instead of letting them embedded?
No that approach won't work. For an intro to what will see the attached. There are many books on robust statistics these days. Best, David Booth
I think what you are looking for is "weighted regression" or regression with weights. The weights can be assigned according to some function of y (but typically one uses estimates of the variance associated with y to account for heteroskedasticity).
And a short description:
http://www.statsoft.com/textbook/statistics-glossary/w#WeightedLeastSquares
"Weighted Least Squares (in Regression). In some cases it is desirable to apply differential weights to the observations in a regression analysis, and to compute so-called weighted least squares regression estimates. This method is commonly applied when the variances of the residuals are not constant over the range of the independent variable values. In that case, one can apply the inverse values of the variances for the residuals as weights and compute weighted least squares estimates. (In practice, these variances are usually not known, however, they are often proportional to the values of the independent variable(s), and this proportionality can be exploited to compute appropriate case weights.) Neter, Wasserman, and Kutner (1985) describe an example of such an analysis".
Omar -
When considering the model part and the 'error' part separately (though the 'error' part is used in the model part), as you do here, this leads to sigma for the error part being described, in statistical learning, as the "irreducible" error. (See Scott Fortmann-Roe, http://scott.fortmann-roe.com/docs/BiasVariance.html. He uses the term "irreducible.")
For weighted least squares, noted by Jochen, the error term is factored into a random factor and a nonrandom factor. The nonrandom factor is the inverse square root of the regression weight, w, explained here:
https://www.researchgate.net/publication/320853387_Essential_Heteroscedasticity. (The idea here, by Ken Brewer, is that variance can be explained by considering every value of y as a cluster of smaller values of y. The within cluster variance is then used.)
We can write that nonrandom factor of the estimated residual as y*^gamma, where y* is a weighted least squares prediction (though in practice, the OLS prediction can be used as a preliminary value which is used to estimate the regression weights), and gamma is the coefficient of heteroscedasticity (see section 2 in the paper linked above). In the simplest regression (no intercept, a ratio estimator) we can use x^gamma. Since w would then be x^(-2 gamma), if gamma = 0.5, then w = 1/x. (Notice that for the ratio model case, y*=bx, and since the relative size is all that counts, a constant b makes no difference, so we can, in that special case, use x instead of y* for a size measure.)
Variance is then a function of predicted y, the model part of the regression. (The attached is from page 2 in the paper linked above, and is for the ratio model case.)
To find the coefficient of heteroscedasticity, gamma, from the data, see https://www.researchgate.net/publication/333659087_Tool_for_estimating_coefficient_of_heteroscedasticityxlsx, and references noted there.
Cheers - Jim
PS -
The weighted least squares definition found through the link Roberto provided mentions a couple of ways the weights can be found. (Also see pages 5 and 6 in https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity.) The one which is not ad hoc, which makes sense from the standpoint of each y-value being a cluster of smaller ones, and is the one I use, is the one described at the statsoft link as follows: "...these variances are usually not known, however, they are often proportional to the values of the independent variable(s)..." However, many people use a subset of those variables as a size measure, but the best one is predicted y itself. Also, the coefficient of heteroscedasticity IS estimable from the data, though accuracy may be low for small samples. Still it is better than assuming gamma = 0 (homoscedasticity), and the "tool" link I have mentions better default values may be used.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression