Hi Matthew

I cam across a paper some years ago that demonstrated that sample weighting in regression analyses was not necessary and made little difference to the results.

Adrian

There may have been some discussion on that on the ASA Survey Research Methods Section listserv some time ago, and I think opinions differ, but it seems a bit redundant to me to use weighting and regression. Perhaps analogously, in the case of missing data, you can weight data to make up for what is missing, or you can impute, and a great way, I think, when regressor data are available, is to use regression. Regression weights are often appropriate, but that is a different kind of weighting.

It sounds like you did some testing to see what looks reasonable. That sounds good, as long as you are sure of what is reasonable.

Hi Matthew, I am not sure what you mean by "exaggerated." Obviously, you feel they are moving in a particular direction. In general when we think about any statistical estimates when think in terms of biased or unbiased estimators. Depending on how the sampling was done and the weighting scheme there should be no reason why the estimates would move in one direction or another. The answer to your question may actually have to do with what software you use. For a long time the majority of software packages did not handle typical survey populations weights properly and SUDAAN was the program of choice for that purpose. I think there are still questions about the use of weights in software and even how they are applied in terms of what the weights need to sum to. Since you are apparently doing logistic regression (odds ratios in your question), there may be more specific issues that arise in your software. You can probably find more specifics by Google searching the name of your software package plus "survey weights" and "logistic regression." Done correctly the unbiased estimates should be those from the analysis using the survey weights, but "done correctly" assumes a lot. From a practical standpoint in most typical designs most people don't see much difference, but some survey designs are such that the results could be quite different. You might have a sense of how your data look. A survey design that fills in a lot of equal size cells and ends up looking very different from a random sample of the population might look different with and without weights. Bob

It is NOT permissible it is OBLIGATORY.

Use PROC SURVEYREG

I believe that you must use the weights in regression analysis on weighted datasets; ignoring this information can often lead to inaccurate point estimates and/or inaccurate standard errors. The fact that survey data were obtained from units selected with a particular - not necessarily simple - sample design must be taken into account in the survey analysis; weights need to be used in analyzing survey data and variances of survey estimates need to be computed in a manner that reflects the sample design. Ignoring the weights would be wrong.

Example through exaggeration: Imagine you had one data point that represented 10,000 people vs. another data point representing 1 person. You wouldn't want to weight those equally, correct?

Of course, weighted data sets can be MUCH more complicated (e.g., there are entire workshops dedicated to understanding the multilevel weighting and sampling plan in the National Assessment of Educational Progress [NAEP], see http://nces.ed.gov/nationsreportcard/tdw/weighting/). Good luck!

Let us close this.

for question;

Is it permissible to perform regression analysis on weighted datasets?

Answer is : because data in weighted data sets are weighted then regression analysis, if you wish to get correct result, MUST be weighted.

why closing? The discussion could get interesting and it is a never ending story to discuss about this topic.

It depends if your model (parameters) describes your population. You may still get reasonable results with unweighted regresssion when this is the case, i.e. it also depends on the distribution of your weights to answer the question how much your results might be biased.

It also depends if prediction quality is of only interest or interpretation is in focus. For the later one you might be on the save side with weighted regression, but not every method is implemented to deal with weights. Then the question remains if your model describes your population well enough.

Mathew -

I am not clear on the application here. There are sample design weights, regression weights, calibration weights (a combination in a sense), and I usually deal with prediction without a sample design, but everyone has different experience, and yours may be an application different than most imagine. Perhaps you are not talking about a survey, which most seem to be assuming.

You might want to review Robert Brennan's comments about logistic regression and about software.

It may be interesting if you provided more information on your application. Perhaps it would be good to just state it as a new question.

Thanks.

What is the sum of the weights in your application? Standard software assumes the weights sum to the sample size (as in "unweighted" data where the weights are effectively all 1). If the weights sum to the population size (as in sample survey data), a crude fix to your problem would be to renormalize the weights to sum to the sample size. A much better approach is to use software designed for sample survey data such as SAS PROC SURVEYREG or PROC SURVEYLOGISTIC. R and STATA have similar procedures. If you do not use weights, you will lose the adjustment you did for recruitment biases. Some would argue that you could avoid weights by adding variables that explain recruitment differences to your model.

There are many ways you can address this "exaggerated" odds ratios. For example, you could divide the weights by the mean of the weights and that will certainly deflate the estimates and the standard errors by conserving the unequal weighting probabilities associated with the sampling design of the data.

Be careful! ... if the weights are "population" weights, rescaling them does not make any sense.

When you have population data that enables you to weight your sample data to adjust for recruitment biases then you have population data that enables you to weight your sample data to adjust for recruitment biases and then you WILL use population data that enables you to weight your sample data to adjust for recruitment biases by using proper method for population data that enables you to weight your sample data to adjust for recruitment biases.

Thanks James...a fuller explanation of my question is as follows: I have comprehensive population data for a period prior to study recruitment. I then commenced with the study, which was a consenting cross-sectional medical research study. Age and gender were broadly representative, but due to unavoidable consenting bias, a certain category of patients were over represented in the sample. Knowing what the population prevalence should be, I created a weighting factor variable to adjust for this, and also a composite weighting factor which also accounted for age and gender for good measure. So in terms of my outcomes, this allowed me to offer an adjusted estimate for prevalence within the population. I was then wanting to undertake binary logistic regression to look at some risk factors which might be associated with my outcomes, and wondered whether this should be undertaken on the new weighted dataset? I hope this is clearer. Many thanks all for your inputs. Matthew

My earlier post was apt. You can use SAS Proc Surveylogistic or similar procedures in Stata, R, SUDAAN, etc. Some would argue that you can avoid weights by adding an indicator variable for your "certain category of paitients" that were over-represented. In the latter case, you will need to pay especially close attention to model fit statistics to make sure the new model fits. If you use standard software (as opposed to software that handles complex samples), you will need to renormalize the weights to add up to the sample size (this can be done by dividing the weights by the sample mean weight).

Stanislav.. many thanks! Answers to questions in reverse order:

2) The study was an autopsy study looking at missed Tuberculosis diagnoses. Inherent with such studies are low consenting rates, as many relatives just don't like the idea of autopsies. We approached the relatives of 90%+ of the mortalities during the study period, but got consent and undertook autopsies on only 15%. The sample was hence primarily dictated by whether or not relatives would consent.

1) The population is specifically 'the population of inpatient mortalities in a hospital'.

My population data comes from a comprehensive 2-month survey of all inpatient deaths that we did before the study commenced. This allowed the collection of basic demographics such as age, gender and admission diagnosis. The 2 month period does not allow for seasonal variations in admissions, but the primary outcome, Tuberculosis, is not seasonal.

Unfortunately I'm based in Zambia.

Then (on top what I recommended above) you are more in the domain of non-response analysis and adjustment. A classic reference is

Groves, R. (2006) Nonresponse Rates and Nonresponse Bias in Household Surveys. http://poq.oxfordjournals.org/content/70/5/646.abstract

Although of course he talks about surveys, frames and populations that are quite different from yours. Nevertheless, the general ideas apply: you need to determine the correlates of response/non-response, and adjust for them. So in your case, the weights are not just "permissible", they are likely necessary.

I agree with Michael Cohen - I would not use weighting when doing a regression analysis and I would have whatever variables went into the weighting as independent variables in my model.

IVEWARE has regression modules for this kind of analysis. You can read about it here: http://www.isr.umich.edu/src/smp/ive/. It's callable from SAS or more simply from the free software http://microsiris.com.

It is possible to use regression models on weighted data. For example, data from survey study are usually analyzed using weighted regression.

I'd prefer to use data as they are...but there may some necessity to balance a-priori. We had a problem of mixing various data types together, each of which coming with pretty different dimensionality. For our research we had to reduce dimensionality of one group as it was going to "take it all". Well, this is not directly applying weights to a regression, nonetheless a matter of a-priori balancing driven by the user's demands. I do not know how to justify this from the viewpoint of a statistical purist, but just passing data to the computer sometimes brings us off-road (?)

I think No need of considering the weight in regression the same result can be found.

If the weights are functions as linear transformation of then the weighing of data will not affect the correlation between the dependent and independent variables. It might on the other hand affect final estimated regression coefficients due to change in level of the independent variables.

Mathew. I ask myself and you: 1) If I use weights and apply them to population does it mean that the sample is not representative? In this case we have two options: a) Get a better sample or b) Accept that analysis is only valid for sample as it comes. And 2) If I predefine weights to variables and sum their product for any purpose, then, am I creating a new variable -as a trial and error function- made of original ones plus weights that must be justified, as also the decision of adding them? I fear that this is a statistical and logical faux pas as you say. It may conduce us to a region of wrong procedures and unreal conclusions. Thanks, emilio

We need distinguish the target population and study population or sample population.

If the sample is not randomly select from the target population, and you want to make conclusion about target population, weigh analysis should be used. Definitely the parameter estimations including odds ratio between weighted and un-weighted analyses should be different. The un-weighted estimation is just for the sample.

In addition, one of the common assumptions underlying most modeling is that each data point provides equally precise information about the deterministic part of the total process variation. The standard deviation of the error term is constant over all values of the predictor or explanatory variables. This assumption, however, clearly does not hold, even approximately, in every modeling application. In such a case, weighted analysis might be also be used to improve the variance homogeneity.

I don't have a concrete answer, but I would think that it might depend on how you weight your sample. For example, if you under-sampled females in your sample (i.e., you know there are approximately 50% females in the general population and your sample only has 35% females), then you could multiply your female subjects by 1.43 to get the required proportion of 50%. This would however create an artificial "identical twin" for 43 percent of your females.

Conceptually, if you do regression on a weighted sample, for example regressing income on years of education, and you are weighting females more strongly than males (because you have fewer females in your sample), then you might be counting female education level with more strength or counting female income with more strength than it really has. I would think therefore that you should not weight your sample when doing regression. That is, do not multiply females' income or education level by 1.43. You could weight the number OF females, but again you would be creating artificial identical twins and all the other factors like age would be distributed according to the females you sampled, not on the actual general population.

From everyone's comments I think the 'weight' of opinion is probably to include the relevant covariates in the regression model and use unweighted data. I think weighting can still be of use though, maybe to give a crude indication of the adjusted prevalence of a certain outcome within the population. Many thanks all for your interesting and insightful comments.

The answer to this question depends on an understanding of what a weighted sample is.

Weighted data corrects for nonproportional sampling of subgroups that have a known probability of occuring in the population. Weights are computed to equalize the effects of over- or undersampled subgroups. This might be deliberate oversampling of an interesting subgroup or the result of sampling error.As long as you know the proportion of the subgroup in the population, you can correct the sample estimates by using weighting.

As an example, suppose you are interested in analyzing variables related to a disease that appears only 1% of the time in the general population. But you would also like to study the subgroup that has the disease, both alone and in the context of the whole population.

Suppose a power analysis indicates you need at least 500 members of the general population for adequate statistical power to estimate relationships in the whole population, but you need 100 members of this subgroup to have an acceptable statistical power for an independent analysis of only this subgroup To get the needed number of subgroup members with a simple random sample, you would have to draw a sample of around 10,000 from the whole population. This is extremely wasteful, as the additional observations beyond the minimum 500 add diminshing ability to detect relationships in the general population (at your desired Type II error rate). Instead you could sample 500  from the general population (which would include an expected 5 from the target subgroup), then sample an additional 95 from the sampling frame of the subgroup. Using these 95 and the 5 from the general population, you have your N=100 for the subgroup analysis.

But what if you combine the two samples to get a representative picture of the whole population? In your combined sample of 595, the subgroup sample is 16% of the observations. In the population, that group represents only 1%. So if you just combine the 95 oversampled observations with the 500 general population observations, you will give each oversampled subgroup observation 16 times the influence that it should have. This extreme bias produces what is technically termed "the wrong answer."

Instead, in the general population analysis, you would weight each observation by the ratio of the actual population proportion to the sample proportion. The weight corrects the influence of each observation from each subsample so that they represent equivalent observations in the population. There are lots of on-line resources that show how to compute the weights. In this example, the N=500 sample would weight each observation by a factor of 1.178 and each N=95 subsample observation by .0626, so that 500*1.178 + 95*.0626 = 595, the N of the combined sample. Now the oversampling bias has been removed in the combined sample.

Using covariates will not remove the bias introduced by nonproportional sampling, as the bias of nonproportional sampling is just as extreme in a covariate as in a any other observed variable. If you have sample weights based on known acccurate population proportions, using them will ALWAYS reduce your estimation error by removing sample bias.

I concur with James. Using " 'weight' of opinion relevant" is not proper. If the sample is not randomly selected from your target population, and you want to make conlusion on your target population, you should use weight analysis to make adjustment. 

Also note that if your sample is nonrandom, your standard errors may be underestimated.  You may need to perform a treatment such as replicate weights to estimate the standard errors.