James R Knaub

James, I sometimes used the weighted regression using Minitab 17 or 18 statistical software package. This procedure in included in Design of Experiments (DOE), in Analysis of factorial design, as indicated below.

Weighted regression is a method that can be used when the least squares assumption of constant variance in the residuals is violated (heteroskedasticity). With the proper weight, this procedure minimizes the sum of weighted squared residuals to produce residuals with a constant variance (homoskedasticity). Determining the proper weight, though, can be a challenging task. Ideally the weight is the reciprocal of the variance of the error. However, this is generally difficult to calculate and other methods are being used. Alternative methods include using:

- The reciprocal of a predictor or squared predictor if the variance is proportional to a predictor.

- Use experience combined with trial and error to see what works.

- Values based on theory, the literature, or prior research.

Suppose your regression model predicts the annual number of traffic accidents in different cities. Because more populous cities tend to have more accidents, the residuals for larger cities also tend to be larger. One method for resolving this is to use the reciprocal of each city's population for the weight.

Caution Weighted regression is not an appropriate solution if the heteroskedasticity is caused by an omitted variable.

In Minitab the weighted regression is included in

Analyze Factorial Design - Weights --> Stat > DOE > Factorial > Analyze Factorial Design > Weights

You can specify weights for your model to perform weighted regression, which is a method to handle data with observations that have different variances

--> Store weights based on the variability in your response across repeats or replicates in Analyze Variability.

Note You cannot specify weights for split-plot designs.

--> . Dialog box items:

Do a weighted fit, using weights in a column:

Enter a column that contains weights for your response. The column must be the same length as the design variables.

Thanks Alexander.

I prefer to use the coefficient of heteroscedasticity as specified in my papers and Ken Brewer's book. As I showed in

https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity,

that method is the one consistent with the format in Brewer's book and dating back to at least the 1930s in agricultural statistics, which Ken showed on page 111 of his book to be expected from an analysis of each prediction as a cluster of smaller ones. (His example was about retail stores, but it really relates to clusters of predictions making up larger ones and the variance within each cluster corresponding to one data point each.) So this shows how heteroscedasticity naturally occurs. (I have also written about other influences.)

Anyway, using a coefficient of heteroscedasticity works well, and with my spreadsheet above, you can select a coefficient of heteroscedasticity, raise a size measure such as a preliminary prediction to twice the negative of that size measure to obtain a regression weight, and enter it into any software which will use that regression weight in a straightforward equation. You obtain that equation by simultaneous equations, minimizing the weighted sum of squares of estimated residuals with respect to each regressor, to find regression coefficients, but why do that if the software will do it? I've worked it out before for one and for two regressors in linear regression without an intercept term, and the huge increase in algebra needed to go from one to two regressors is astronomical. But no problem when you can use software such as SAS PROC REG and enter the regression weight, w.

But I would like to know examples of software that will let one enter w and then use it in the polynomial form of linear regression. I don't think SAS PROC REG will do that, but maybe SAS PROC GLM. What about other software. R for example. I understand you use Minitab. What about that?

Sorry it was not clear, but that was my question.

When you use a coefficient of heteroscedasticity in a regression weight expression, the prediction 'formula' and estimated variance of the prediction error, which can be used in prediction intervals, work out well for prediction and analysis. I have plotted prediction interval bands around prediction lines on scatterplots. The same would be possible for polynomial linear regression for any software which has solved for minimizing the regression coefficients.

I just want to know what software doing polynomial regression will let one enter a regression weight.

Will Minitab do that?

Thanks.

All - I just want to know what software doing polynomial linear regression will allow you to enter a regression weight. - Thank you.

James R Knaub

1. " But I would like to know examples of software that will let one enter w and then use it in the polynomial form of linear regression... maybe SAS PROC GLM. What about other software. R for example? I understand you use Minitab. What about that? "

James, I did not use R for weighted regression, and I do not know of an R code for doing this.

2. " I just want to know what software doing polynomial regression will let one enter a regression weight. Will Minitab do that?"

As I mentioned earlier, Minitab allows doing weighted regression but not higher than a 2-nd order polynomial. You have to enter weights in a separate column corresponding to the column with responses. I guess that it is not a most convenient way for doing a general weighted polynomial regression but I do not know a better piece of software.

Dear James

I think that Proc REG or GLM of SAS will not do that because you need to ajust your weights iteratively and they lack programming facilities, but there are several SAS Procedures that may obtain it because in them you may program the change of the weights. For Example in SAS/STAT: the Classic Proc NLIN (https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#nlin_toc.htm), or Proc NLINMIX (https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#nlmixed_toc.htm), Proc GENMOD (https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#genmod_toc.htm) or Proc GLIMMIX (https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#glimmix_toc.htm) and Proc NLP from SAS/OR (http://support.sas.com/documentation/cdl/en/ormpug/59679/HTML/default/viewer.htm#nlp_sect4.htm)

First, the difference between the weight adjustment some are used to, and what is done to find the w I would enter here is spelled out at the bottom of page 5 and page 6 in  https://www.researchgate.net/publication/333642828_Estimate_Coefficient_of_Heteroscedasticity, where

you can see a comparison of methods. I show why the method I use is better. It is a well-known format that has been discussed for many years by many experts, and Ken Brewer used it in his book to do what I have also done for decades, but in his book he also proved limitations on the naturally occurring heteroscedasticity. He was an expert in this area even longer.

Second, there are only two runs needed, besides my spreadsheet. The first one gets OLS preliminary results to use for size measures, as I have noted in papers referenced above. You then can use my spreadsheet to get w. All I want is to know which software run polynomial linear regression and accept an input for the regression weight, w.

For many years the US Energy Information Administration has run SAS PROC REG with an input for w for linear regression for tens of thousands of cases a year. I just want to know the software that handles polynomial linear regression.

All of the objections above would be seen to have been completely unnecessary by reading the examples in my linked papers.

This works and has worked well for some time.

So,

All - I just want to know what software doing polynomial linear regression will allow you to enter a regression weight. - Thank you.

Dear James. Any of the procedures that I mentioned would allow polynomial regression or non lineal regresion with simultaneous reweigthing of its observations, but then the gamma coeficient would be other parameter to estimate with the usual beta0, beta1, beta2.... and sigma.

If you want to use a non iterative aproach that estimates beta0, beta1, beta2 and sigma with a weight introduced in the same data step, you may use PROC GLM. In this procedure you define that a variable is classificatory by the use of CLASS sentence, and any continuous regressor by excluding them of the CLASS sentence. Eg.

PROC GLM DATA=DATOS;

MODEL Y= X X*X X*X*X;

will estimate a 3rd grade polynomial because the '*' operator creates the variables X2 and X3;

and

PROC GLM DATA=DATOS;

CLASS Trat Block;

MODEL Y= TRAT Block X;

will estimate a Covariance model with X as covariable in a Complete Random Block Desing.

____________

In PROC REG you may obtain a similar result if you compute X2 and X3 with adecuates variable names in a previous DATA STEP (before the PROC REG).

e.g.

DATA DATOS;

set DATOS;

X2=X**2;

X3=X**3;

output;

run;

PROC REG DATA=DATOS;

MODEL Y= X X2 X3;

run;

where you computed before X2=X**2; and X3=X**3;

The PROC REG has several plots available into the procedure and you may run several models, reweigth the observations and run the same or other models in several steps within the same run of the procedure.

I hope this help.

OK. Let's keep this straightforward. Looking at the attached expressions for this question, you can see that the regression weight is a size measure (the predicted value would be best, but a preliminary OLS value is fine and practical) raised to twice the negative of the coefficient of heteroscedasticity. (If you have the simple model y*=bx, you don't even need to do a preliminary run because you can just use x as the size measure. Multiplying by any constant will make no difference.)

Since 1999, I've had the US Energy Information Administration use SAS PROC REG for production runs of Official Statistics where I proposed a robust value for the coefficient of heteroscedasticity to go into w, based on an analysis of the data. This has worked very well.

I am retired now, and recently looked into making this available to others by writing a paper on estimating or selecting a default for the coefficient of heteroscedasticity, with examples, and a spreadsheet tool to go with it. Once you decide on the coefficient of heteroscedasticity to use in that weight, you can run SAS PROC REG with that weight, w, and results are now WLS results, rather than OLS. (Note that for OLS, the coefficient of heteroscedasticity is zero. We just often want to use a better default than zero.)

So, SAS PROC REG works fine for simple and multiple regression. Just enter w, as shown in the that attached figure in the question above. But in place of the weighted prediction for a size measure in that expression for w, we use a preliminary prediction. All of that is shown in examples in my "Estimating the Coefficient of Heteroscedasticity" paper, and the spreadsheet, as I've noted above. None of this should be a problem for anyone who reads that paper.

So ... SAS PROC REG works well for the usual simple and multiple linear regression. Just use what is in that paper and spreadsheet to provide an expression for the regression weight, w.

SAS PROC REG with w entered has been used extensively. No problem. And this should work well with other software packages.

What I'm asking about is software for polynomial linear regression. I am retired now, and have no way to find out about software capabilities other than documentation online, and asking here. I looked to see if SAS PROC REG with a w entered could be used for polynomial linear regression, and it appeared that the answer was no, but that SAS PROC GLM would do this for quadratic linear regression, allowing w to be entered. But being retired I could not find out except by asking.

I am asking about any and all software packages that will do this.

.

.

.

So, Guillermo, are you saying that SAS PROC GLM will let one enter regression weight w on a single run (once you have a w to enter), and do quadratic linear regression? That is what I thought, but was hoping for confirmation. Yes or no.

.

.

.

All - I just want to know what software doing polynomial linear regression will allow you to enter a regression weight. - Thank you.

Dear James

I think that YES, GLM would do this! If you have an example I could try to program it!.

All - I just want to know what software doing polynomial linear regression will allow you to enter a regression weight. - Thank you.

By the way, if you had no packaged software and had to program this yourself or get someone to do it ... back when one might have done this using Fortran or SAS basic programming ... you'd have to derive the 'formulas' yourself, but at least then you know what it all means.  Here is a case where I derived this for two regressors with no intercept term:

 https://www.researchgate.net/publication/261534907_WEIGHTED_MULTIPLE_REGRESSION_ESTIMATION_FOR_SURVEY_MODEL_SAMPLING

Starting at the right column of page 597, you will see how to minimize Q to get the regression coefficients for WLS regressions.

Guillermo -

Thank you.

One of the examples in my "Estimating the Coefficient of Heteroscedasticity" paper is a polynomial regression. The data I'm told came from a US Govt project, so that is why the data are publicly available. In my paper,

https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity

this example starts on page 18. The data are noted as coming from a reference. Here is that reference:

Frost, J.(2019), Statistics by Jim, “Making Predictions with Regression Analysis,”

https://statisticsbyjim.com/regression/predictions-regression/, found under “Regression Tutorial with

Analysis Examples,” https://statisticsbyjim.com/regression/regression-tutorial-analysis-examples/,

downloaded May 23, 2019.

Thanks - Jim (Knaub)

("Statistics by Jim" is by Jim Frost)

PS - Guillermo - If you are having trouble finding that data file, it is under

https://statisticsbyjim.com/regression/predictions-regression/,

and looks like this: "You can download the CSV data file: Predict_BMI."  It is found part way down, a little before  the heading "Finding a Good Regression Model for Predictions." After a couple of paragraphs below that heading is the scatterplot showing the quadratic regression which I reprinted, with permission, in "Estimating the Coefficient of Heteroscedasticity."  On the top of the scatterplot, you see %Fat = - 23.19 + 3.286 BMI - 0.03999 BMI^^2.  This should be the result when homoscedasticity (gamma = 0) is assumed.  That is, the coefficient of heteroscedasticity is zero for this result. 

At the bottom of page 22 in "Estimating the Coefficient of Heteroscedasticity," you'll see I suggested using gamma = 0.7 for this case.  That gives us

 w = z^( - 1.4), where z is the size measure.  Here the best available size measure is the OLS prediction from the first run.  (The predictions from a preliminary run without entering w, and thus assuming all weights are equal, have to have been saved for this.)  That should be close enough, as the coefficient of heteroscedasticity itself is a rough estimate.  However, we know that quite often, a coefficient of heteroscedasticity of zero is used, and that is often far from optimal, so we've made an improvement. 

(The only time a preliminary run is not needed is for the simplest of practical models, y = bx + e.) 

This would yield the improved, now WLS predictions.  However, they should not change much at all here.  But, the estimated variances of the prediction errors, and thus the prediction intervals, would change substantially, at the high and low ends. 

.

.

.

.

But, all I asked was to know which software would run polynomial, or at least quadratic linear regression, and accept a regression weight input.  If the above is a lot of bother, and without the incentive of solving a problem you need to solve, I'd say it was a lot of bother, you could just run once, and put something in for w such as w = 1/x, just to see if it works. 

.

.

All I asked was to know which software will run polynomial (or at least quadratic) linear regression, and accept a regression weight.  So the main thing here is to know, in this case, will SAS PROC GLM do that? 

Thanks. 

.

.

.

Does anyone else know what other statistical software will run polynomial linear regression (or at least quadratic linear regression) and let you enter a regression weight?   Thanks. 

Dear James

I verified that I could obtain your results with GLM. I adjunt the outputs.

The SAS Program would be:

*fatp is the data file that you used;

proc glm data=fatp;

Title '%Fat predicted from BMI';

model _Fat=BMI BMI*BMI;

output out=predicho p=pred r=res;

run;

.....

data predict_bmi_b;

set predicho;

drop pred res;

w=pred**(-2*0.7);

run;

proc glm data=predict_bmi_b;

Title '%Fat predicted from BMI after weighting with g=0.7';

weight w;

model _Fat=BMI BMI*bmi;

output out=predicho_b p=pred r=res;

run;

The same results can be get with PROC REG (If you wnat I can send them), computing BMI**2 previous to the first run.

What I could not appreciate yet is how the analysis using WLS is better than the previous OLS analysis for this data because I do not noted an important lack of homogeneity. On the last graphs we can see the residuals and predicted of both models that are almost aligned on a identity.

Dear James

I tried PROC NLMIXED that allows to estimate gamma with the rest of the parameters and I got convergence with gamma=0.4984 rather similar to your gamma 0.7 (If you want I can send them )

Guillermo -

Thank you for running the PROC GLM.

You said "On the last graphs we can see the residuals and predicted of both models that are almost aligned on a identity." But remember that when the data are symmetric like this, I said the following above: "This would yield the improved, now WLS predictions. However, they should not change much at all here. But, the estimated variances of the prediction errors, and thus the prediction intervals, would change substantially, at the high and low ends."

The main difference between OLS and WLS results is with the variance of the prediction error. The predictions are usually close, but it depends. OLS should be "unbiased" but "inefficient."

So, your results were as I predicted above from looking at this particular scatterplot. In fact, because the curve runs almost parallel to the x-axis, this will dampen variance differences. But it was the only polynomial regression example I had. I haven't worked with them, but wanted to know which software would work for those times someone would need it.

Thank you very much. This was very helpful.

Cheers - Jim

Dear James

I computed with GLM the "Standard error of the individual predicted value", without weighting (gamma=0), and weighting with gamma=.7.

I adjunt the graphs that as you said are very different but which is better and why?

Sorry. I can't see your graph with my phone. But the WLS estimated variances of prediction errors should be smaller for smaller predictions, larger for larger predictions. The OLS estimated variances of prediction errors are the same, regardless of prediction size. Thanks.

Regarding which is better and why, WLS is better, because the data themselves can tell you how much variance increases with size.  The data tell you to use WLS regression.  Graphical residual analyses are very useful, and can be used as the basis for estimating the degree of heteroscedasticity.  You might obtain a rough estimate of degree of heteroscedasticity, but that's generally better than assuming there is none, and my paper "Essential Heteroscedasticity" addresses Ken Brewer's assessment of the expected range of the degree of heteroscedasticity.   (However, heteroscedasticity is also impacted by model specification and data issues.) 

Generally speaking, OLS makes no sense to me.  I would generally not expect homoscedasticity.  For something a million times bigger to have the same variance seems unrealistic.  The theory for heteroscedasticity that I like is the one in Brewer's book, page 111, which does not specifically mention cluster sampling, but obviously relates to it.  The idea is that each prediction is like a cluster of smaller ones.  The within cluster variance determines the level of heteroscedasticity. 

I have an example in mind for another paper.  Well, I have more than one, but the one I'm thinking of now looks like you should use OLS, and as I recall, the data even show that to be the case, approximately, but I can see that the reason is that the predictor that was used in that example I found on the internet,  conflates other information.  I have come to the (tentative) conclusion that when heteroscedasticity does not fall in the range that Ken Brewer described, it is because of model specification and/or data issues, such that the best models are not only unbiased and with lowest variance, but also have a coefficient of heteroscedasticity which is in the range Ken Brewer described. 

Anyway, the WLS results are best.  They will often be close to OLS results for the predictions, but less so for many of the variances of the prediction errors. 

OLS is just a special case of WLS, and usually not the best.  When using OLS, we are assuming a coefficient of heteroscedasticity of 0, which is generally not the case.  The data tell us this. 

I read an argument previously that I think indicated, as I recall, that if we can't say very well what degree of heteroscedasticity we have, we should use OLS.  No, that makes no sense.  If you think the coefficient of heteroscedasticity is somewhere between, say 0.6 and 0.9, why then make it 0? 

So your best WLS regression is the best you can do.  

Two notes above I said "The OLS estimated variances of prediction errors are the same, regardless of prediction size." Actually I think that works out to "about the same." But yes the estimated variance of the prediction error for weighted least squares regression for a much larger prediction tends to be substantially larger.

Thank you very much to Guillermo Enrique Ramos for confirming that SAS PROC GLM can be run for quadratic regression with a regression weight.  That regression weight can be entered in the attached format, as will be true for any regression of the form y = y* + e for software that accept regression weights. Those weights are used when minimizing the weighted squares of the estimated residuals for each regression coefficient.  

The best size measure, z, would be the best WLS prediction, but since that would not be available, the OLS predictions will be OK for size measures here.  (OLS regression is a special case of WLS regression. The predictions are often close.  The variances of the prediction errors generally are not very close.)

SAS PROC REG with regression weight w, works well for other linear regressions, referred to as simple linear regression, and multiple linear regression.

.

.

What other software does this?  I mean, you just enter w to go from homoscedastic to heteroscedastic regression.  Perhaps R will do this?  Stata?  -  Thank you. 

Proc Reg may be usted without problem. It only requiere some trivial programming.