Just as you said, in most cases, when the predictor x becomes larger, the variance of y will become larger. Suppose the equation is y=f(x). We presented a generalized weight function w=1/f(x). In most cases, it works well. More general, it can be expressed as w=1/f(x)^n, n=0.75-1.0. It is very similar as your gamma between 0.5 and 1.0. I hope it is helpful.

Wei-Sheng Zeng - 

I like Fugure 1 in your first attachment for demonstrating heteroscedasticity in nonlinear regression. 

It is interesting that the same regression weight function is useful in forestry, establishment surveys, calibration of lab equipment, I think maybe hydrology, and I suspect many other areas. It is also interesting that this can be developed independently in these various areas, like many other ideas, that we may then find to be complementary.  That is, I like to see how these ideas develop in different contexts and how we can learn from this.  

Any other examples of any kind where x and or y are often negative?  I thought there might be some in biostatistics, for example.  I am particularly interested in cases where there can be negative x and/or y, especially if they have relatively large magnitude. 

Jim

Jim,

In my field of laboratory analytical chemistry, amounts are always positive, and we tend to set zero to zero amount. I can't think of any useful modern instrumental methods where x is the rate of change of amount or concentration.

Presumably there do exist process monitoring applications where the signal, for example downstream of a blender, is the density, refractive index, optical absorbance or rotation, etc. with respect to a reference cell containing the mixture at its target composition.

Ratios of stable isotopes, such as in ice core samples, are always determined with respect to agreed reference materials, because you can't directly calibrate a mass spectrometer with sufficient precision and accuracy. That means both the 'delta' and the corresponding ratioed signal can be positive and negative with respect to the reference point. This is a long-standing, complex and well-studied field; non specialist mass spectrometrists like me keep themselves at a respectful distance.

Finally, in electrical systems with reactive components (capacitors, inductors and their electronic emulations), induced currents and voltages are bipolar, one being a function of the rate of change of the other. Perhaps there exist relevant publications on, for example, the characterisation of dielectrics and magnetic materials.

My 2 cents: I also can not imagine any example where there are negative and positive values, together with heteroscedasticity. Heteroscedasticity is mostly a sign of a "wrong" statistical model, e.g. when an additive model is used to model actually multiplicative errors. A reason for getting negative values in such cases (where the response actually must be >= 0) may be a "wrong" background correction (often a subtraction). Instead of trying to find a statistical model that describes the resulting data correctly I would rather go for a better way to correct (or account for!) the background.

In linear regression through (really to)the origin, it does not make sense not to naturally have heteroscedasticity, and thus one may use weighted least squares regression.   I am not so sure of other cases. 

Jochen - Are you referring to cases where relatively small magnitude negative values occur because of data definition problems?  I have seen that, say when net rather than gross generation was used in an electric power survey. There I tried to get people to look at gross generation. - Jim 

I thought there would be some applications of regression going substantially through quadrants, other than primarily the first quadrant.  Perhaps in biostatistics?  I just added biostatistics to the "TOPICS" above.  

James, I have very simple examples. One is the determination of the concentration of some soveld analyte by light absorption e.g. the UV spectrophotometry of nucleic acids.

Light of a given wavelength (e.g. 260nm) and intensity I0 is passed through a cuvette and measured by a detector. The log of the relative loss due to the precence of the analyte (e.g. DNA) is defined as the optical density:

OD = log (I0/I)

When 100% of the light passes,the OD is log(1/1) = log(1) = 0

When 10% of the light passes, the OD is log(1/0.1) = log(10) = 1

Within a "window of linearity" the OD is proportional to the concentration of the analyte, i.e.

c = p*OD

where c is the concentration and p is a proportionality factor.

In experimental setups the analyte is not the only substance absorbing light at the given wavelength. Typically there are buffer components or contaminations that affect the measurement. In some cases the absorption of the buffer can be higher that that of the analyte.

The typical solution of this problem is to subtract the OD value of the buffer ("blank") from the OD value of the sample.

Depending on c and on the size of the measurement error, the resulting ("blanked") OD can be negative. Additionally, the error laws for c are Poisson (for going down to a few molecules), gamma, or log-normal. The error of the OD measurement is further related to c (higher c -> less light passes -> lower signal, higher relative noise). If a dilution series is measured to get a (linear) calibration curve, the curve should go through the origin, there can be negative values at low c, and the error of OD is heteroscedastic. The "faulty" procedure of blank-correction will also lead to a loss of linearity. However, the effects and drawbacks are typically negigible. It's nor an academic example.

I would party agree with Jochen's idea, with different words: in many (all ?) biological applications, the interesting variable cannot be negative, hence a Gaussian distribution is not applicable (in theory) but a log-normal one is. And in that case, variance is "naturally" (almost) proportionnal to the mean --- heteroscedasticity is a consequence of the log-normal distribution (I would not say it is a consequence of a wrong model, but's that a question of point-of-view --- as, if the distribution is Poisson, it also gives to natural heteroscedasticity).

As said by Jochen, the easiest solution in that case is to log-transform the data, to have homoscadsiticy and use OLS,  and express the results in a multiplicative scale...

Both logarithmic regression and weighted regression could be used to eliminate the influence of heteroscedasticity in the data, and if the model from logarithmic regression was corrected properly, it would be almost the same as that from weighted regression. More detail could be seen in the attached paper.

Thanks All.

I also can only think of regressions that are generally in the first quadrant, that is, where y and x (or y and all regressors) are nonnegative. But I feel certain that at some time I have seen graphs that went through other quadrants, and regression applications there, I thought, might be the reason for the popularity of OLS. In the case of linear regression through the origin (well, from the origin), it does not seem that there would ever be homoscedasticity.

Even if there are good applications for regression in other quadrants, it would seem that heteroscedasticity would exist in some form related to absolute values. Does anyone have such an example?

Note: I consider heteroscedasticity to be an important part of the error structure, not something to try to remove. (Removal may be desired for purposes of conducting hypothesis tests, but as a p-value is a function of sample size, I would avoid them in favor of simply comparing parameters to their standard errors.)

Anyway, I am asking for a regression application that does not feature the first quadrant.

Thank you - Jim

You may see this example as more a technical detail than a real one, but... In several applications, when you expect a linear relation after log-transformation of the data, you can very well have both positive and negative values for Y or X and Y if the original value is below 1.

Two typical applications would be

- monoexponential decay, for instance 1-compartment model in pharmacokinetics;

- allometric relation (Y = a * X^b )

Both cases however could also be seen as non-linear regression in the first quadrant,  so I'm not sure it is what you are searching for...

Another (real) example: in quantitative real-time PCR, amplification efficiencies are determined by the slope of the linear relationship between the logarithms of the (known) dilution factors of a standard and the (measured) ct-values.

NB: Sometimes the concentrations are used instead of the dilution factors (when the concentration of the standard is known). The concentrations are usually measured in molecules per reaction what is >1, so logs of the concentrations are positive. Because of the log-linear relation this does not change the result. When the standard concentration is set to 1 (arbitrary conc. unit), the logs of the concentrations of the diluted reactions are again negative.

I don't think heteroscedascity (or homoscedascity) is affected by the sign of the variables: heteroscedascity depends on the variance, and the variance of a variable does not change when you add or substract a constant to the variable. So, if you transform your original variables adding a constant to them, something like: Y' = Y + K; X' = X + L , K and L being constants. You can choose K and L so that both X' and Y' have non-negative values. And their variance is not affected, i.e., Var(Y) = Var(Y') and Var(X) = Var(X'). If the variances of X and Y have not changed, neither has heteroscedascity.

Javier, you have an interesting point. (I think you meant "negative," not "non-negative.")

However, l really thought that in some discipline/subject matter, someone would have an example application where most or all of the data could be inherently negative.

I broadened the "TOPIC" list to try to include more areas of research, in an effort to find such examples. If there are no such cases, much to my surprise, then it seems that every regression is either a regression through the origin, linear or nonlinear, or a somewhat displaced version of such a regression. Considering linear regression through the origin as one visualization, if x were, say, generation capacity of a windpowered electric generation plant, from a census and y were the corresponding generation for the corresponding plant in a given monthly sample for some official statistics, then a regression through the origin could be used to 'predict' the missing (out of sample) data. In such a case, it might be reasonable that a predicted y of 500 megawatthours may have a standard error of 10 megawatthours, but would it also be reasonable that on the same regression line that a predicted y of 40 megawatthours also have a standard error of 10 megawatthours? No, that does not happen. And I cannot think of any application of regression through the origin where I would not expect substantial heteroscedasticity. If every application of regression is just regression through the origin, or some displaced version of it, then why would OLS ever be useful? I've been told before that it is a good default if you do not have an estimate for the regression weights, but it is always possible to estimate a coefficient of heteroscedasticity, and in my experience, equal weights would often be a very bad default. So why ever use OLS?

I still imagine that someone may have an application/example of regression where OLS (homoscedasticity) may be useful, and I conjecture that it may involve negative numbers that are not just displaced positive ones.

Does anyone have some experience to share, regarding this? 

In Economics, we often have examples with negative values... for example, let's imagine a model where we want to analyze the growth rates of countries. Some of the explanatory variables might be the growth of the employment rate in the country or the trade balance. All three variables can perfectly have negative values, and I do not consider that there is a problem with that. When you work with growth rates, inflation, real interest rates, trade balances, payment balances, fiscal deficit or superavit... it is not unusual to have both positive and negative values within the same variable.

By the way, the electric generation example above from establishment surveys of finite populations is of course a limited area. In some regression applications there may be substantial interest in something else, such as influential points. But I would like to hear about heteroscedasticity or homoscedasticity, as applied to regression that is not primarily in the first quadrant. Thank you. 

PS -  Ah. Just got your note above, Javier.  Thank you.  So, do you know something about the heteroscedasticity in such cases perhaps? 

On the risk of repeating myself: multiplicative models the logs can have a linear relationship that can be modeled by linear regression. These logs can be negative, the regression line may or may not go through the origin, and typically the errors (on the log scale) are homoscedastic. The real-time qPCR data was one example.

You may respond that only untransformed data should be considered. But this is not a good argument (IMHO), because our "linear" scale has no natural preference over any other scale. Again coming back to the qPCR example: the measured values (ct values, fractional cycle number required to amplify an intial amount to a fixed measurable signal) are proportional to minus the log of the concentration. Which scale is to be preferred? The scale of the concentration or the scale of the measured variable (cycles)? Both ways are correct; only using "cycles" (i.e. use the log sclae of the concentration) makes the math simpler.

Javier - In case you were not looking at regression weights, one can consider a regression weight with one regressor, x, as x^(-2gamma), where gamma equal to 0.5 or maybe a little larger, may be robust, if you have not estimated it, or need to underestimate a bit due to data quality issues, perhaps. For multiple regression, the size x could be a predominate regressor, but if you have the resources, I like to use a preliminary predicted y for that, say the OLS one. - Jim

Jochen - Thank you.  -  I understand that the application you mentioned is not 'completely' linear. That's OK.  I was looking for all examples.  -  By the way, are you sure about that homoscedastic part?  - Jim  

Jochen is right about the logarythms: in Econometrics, it has been empirically tested that the logarythmic transformation reduces (or even elmiinates) heteroscedascity. And it solves also the "multiplicative" problem, because if the original model is: Y = X·Z , the transformed model would be: log(Y) = log(X) + log(Z) (nd, thus, is a linear model).

Another useful transformation, if the variables are measured in very different scales is the standardization of the data: X' = [X - Mean(X)] / Standard Dev. (X) . This eliminates the "scale" problem, although you will have negative values (those lower than the variable mean) with a 100% probability. This transformation is often used in Econometrics.

Thank you.  Good point. 

I like Carroll and Ruppert (1988), Transformation and Weighting in Regression, though that is not an econometrics reference. 

Nice responses, but I still hope someone has an example, untransformed, that has even less to do with the first quadrant.  I just thought something like that would be of interest, not that the responses given were not interesting.  They were. Thanks. 

In one of Javier's comments, he mentioned this:

" When you work with growth rates, inflation, real interest rates, trade balances, payment balances, fiscal deficit or superavit... it is not unusual to have both positive and negative values within the same variable."

When I read that, I was thinking in terms of multiple regression, and if you put several of these variables together, you could make a two-dimensional plot of y vs predicted y, which might be primarily in the first or third quadrant, and then, perhaps somewhat related to another one of Javier's comments, the heteroscedasticity in the third quadrant might be somewhat like a mirror image of that in the first. (I refer to Javier's comment that changing the sign of y does not change the variance of its prediction error - though the context there was different.)

But looking at Javier's comment quoted above more closely now, it looks like there is more potential for the kind of examples about which I was most curious when writing the question.

(On a side note, I encourage everyone to look at the first figure in Wei-Sheng Zeng's first attachment. It is a nice illustration of some other things that came up here, to some extent.)

May be searching for something related to temperature (expressed in usual scales, like °C, at least) would do the job, at least on one of the scale? But biology may not be the good domain in that case...

Another idea could be chemistry with predicting some chemical properties based on pKa values (even if it could be argued that pKa = log(Ka) is already a transformed data... pK are much more used, again a matter of convention) - something like Hammett's sigmas

http://en.wikipedia.org/wiki/Hammett_equation?oldid=402181218

Note however that a strong part of « first quadrant » may come from the fact that lot of domains like positive values, first, and second strongly depends on the scale choosed (think about the temperature example, °C or K), so may be not allowing transformations or changes modeled by covariates (and not absolute variables) is a little bit « overkilling » ?

I come at this from a rather different perspective - that of random coefficient or as it is sometimes called multilevel model. There it is quite natural to specify and model a variance function that depends on explanatory variables.

Imagine you have pupils at level 1 nested in schools at level 2 and the response is a Maths score now and the predictor is the Maths score 3 years earlier on entry to the school. In the fixed part of the you would have the general mean line across all pupils and across all schools. At level 2 one could have (what is known as a random intercepts and random slopes model) such that there are bigger differences between schools in progress for children of lower ability on entry - that is your are explicitly modelling the between-school heterogeneity of the earlier Maths score.

Much less well- known is that you can also simultaneously model within-school, between pupil heterogeneity. Thus, the lower ability pupils may be more variable in their progress - this seems perfectly natural to me. It is important to do this as it is substantively interesting and you can get confounding of heterogeneity across levels, so the between variance can be mis-estimated when it is really within heteogeneity. 

The gamma model is a general procedure that effectively works on the response and all the predictors (constant coefficient of variation model so that the variance is a function of the mean). In that regard it is like a logit/Binomial model that has inbuilt heterogeneity with greater heterogeneity expected when probababily of positive outcome is around 0.5, but narrower variation as the probability gets closet to 0 and 1 - heterogeneity is an integral part of the estimation of such models. 

What I am talking about however allows differential heteogeneity for different predictors and it is even possible to have  differenential heterogeneity for categorical predictors - such as differential variability for boys and girls in their progress.  

In such work, heterogeneity is expected and is not an aberration. eg you can have impact heterogeity where two drug treatments have the same mean lowering of blood pressure but one is more variable in its effect than an another. Of course you would try and account for this differential impact but is also useful to find that it is going on.

Here is a paper on Rgate that is a tutorial on these type of models

https://www.researchgate.net/publication/23538539_Modelling_complexity_analysing_between-individual_and_between-place_variation_--_a_multilevel_tutorial

and here is a training manual for MLwin that can fit such models  with complex variance functions at any level

https://www.researchgate.net/publication/260771330_Developing_multilevel_models_for_analysing_contextuality_heterogeneity_and_change_using_MLwiN_Volume_1_%28updated_June_2014%29

Article Modelling Complexity: Analysing Between-Individual and Betwe...

Book Developing multilevel models for analysing contextuality, he...

In the case of your establishment survey data, I would strongly advise to work on the log of establishment size (and of some explanatory variables). This would limit, and even probably fully solve your heteroskedasticity problem. You may be interested in the results we obtained in the French case, available in the following report.

In order to solve the problem of zero's inside the log, if any, you may compute log(1+size).

Book Etude des décisions de localisation des emplois en Ile-de-France

Nathalie - 

I do not see heteroscedasticity as a problem, but as a natural part of the error structure. That is the purpose of weighted least squares (WLS) regression. Determining a coefficient of heteroscedasticity has worked very successfully for me for many years. It can be estimated several ways, but generally it seems that underestimating it is more robust.

A number of people have worked in this area. Sarndal, Swensson, and Wretman in 1992 have a well-known book and Holmberg and Swensson wrote a very relevant paper in, I think 2003. Ken Brewer's book of 2002 is good, and I have many papers on this on my ResearchGate page, with many references. There is a good book, Transformation and Weighting in Regression by Carroll and Ruppert (1988) that I like, but unnecessary transformations just 'muddy the water' in my opinion. 

The first person to respond to this question had something to contribute on this that you may want to read. 

Jim

The log transformation does not "just muddy the watter", but it has many strong advantages, as already noticed by other people in this discussion. This is NOT an unnecessary transformation, but a necessary and strongly advised one in the context of the original question. One of the main advantages is that it leads to multiplicative rather than additive effects, which is by far more plausible in the context of size effects in establishment survey data.