I am only a student and thus may not have great insight in methods like other contributors here but, i would strongly advise to look through

"Applied Survey Data Analysis, Second Edition Steven G. Heeringa, Brady T. West, Patricia A. Berglund" Examples of similar problems are shown with interpretation of results and assessment of the standard errors. all you need to do is identify the design variables ( PSUs, Final weights, stratum variable) these are always provided by the data provider if you read the documentation. once you declare these, the computational models are the similar and depending on the software of choice, you will obtain design adjusted standard errors with adjusted degrees of freedom.

This is a useful practical account of the issues

Applied Survey Data Analysis, Second Edition

Steven G. Heeringa, Brady T. West, Patricia A. Berglund

SPSS does have the capability for the analysis of complex samples

http://www.sussex.ac.uk/its/pdfs/SPSS_Complex_Samples_22.pdf

And this will give you some insight into how the survey has been designed

Article The Behavioral Risk Factors Surveillance System: Past, Prese...

The golden rule is that what goes into the design must be taken account of in the analysis.

Thus with a multistage design, the respondents in the PSUs tend to be more alike (dependent) than a simple random sample. You have not get as much information as you think you have. Small amounts of dependency can have big effects on the standard errors which are estimated to be too low resulting in increased risk of Type 1 error. To resolve this you can use some sort of robust procedure to inflate the SE, or you can use mixed or multilevel models to model the dependency explicitly and thereby correct the standard error. You might also need to use sampling weights if certain types of people have been over-sampled and you are interested in the prevalence rate. States appear to have been used as Strata ( to get a sample from many States even ones with small populations) so that also should be taken into account.

If you are struggling , find someone to help they are often located in epidemiology or biostats departments.

This is a concise introduction

Analysing Complex Survey Data: Clustering, Stratification and Weights by Patrick Sturgis

http://sru.soc.surrey.ac.uk/SRU43.PDF

I think I would look for a paper that has addressed the issues associated with the survey design as well as pursue the route suggested by Kelvyn Jones above. At the back of my mind would be to use an alternative, less complex, data source. But I don't know if that is feasible in your case.

Thanks Kelvyn and Mervyn for sharing your thoughts and references. I'll take a close look on your comments and suggested references to address the reviewer's comment.

Hello Prof. Kelvyn:

I have one additional question about your comment. Here was your comment on my question in below:

"Thus with a multistage design, the respondents in the PSUs tend to be more alike (dependent) than a simple random sample. You have not get as much information as you think you have. Small amounts of dependency can have big effects on the standard errors which are estimated to be too low resulting in increased risk of Type 1 error. To resolve this you can use some sort of robust procedure to inflate the SE, or you can use mixed or multilevel models to model the dependency explicitly and thereby correct the standard error. You might also need to use sampling weights if certain types of people have been over-sampled and you are interested in the prevalence rate. States appear to have been used as Strata (to get a sample from many States even ones with small populations) so that also should be taken into account."

In your comment, you suggested to use mixed or multilevel models to model the dependency and correct the standard error. You also suggested to use sampling weights for population (actually, one of the reviewers also commented me to perform weighted least squares (WLS) regression that accounts for population size).

Prof. Kelvyn, my question is how we can incorporate both mixed model and WLS together into one model. I'm currently using SPSS to run both mixed model and WLS seperately, but I have no clue on how to incorporate both mixed model and WLS into one model (In SPSS, mixed model and WLS are under different menu).

Please advise on this. Thanks again!

Mervyn's advice is good - have a look for a published paper on the BRFSS in a high quality journal.

I think it needs stressing that the design is not complex for its own sake - most professional surveys will be designed as such to be represenative and efficient.

Now to answer the question. I should say at the outset that I know nothing about the BRFSS nor enough about SPSS to be useful.

We are here talking about sample or survey weights and these if relevant should be made available by the survey collectors and my first google got this

https://www.cdc.gov/brfss/annual_data/2014/pdf/weighting-data.pdf

If I was using multilevel modelling to analyse these data I would use a thee level model of individuals nested in PSU's nested in States and I would use individual weights at level 1 and PSU weights at level 2 if they exist.

Here is some training material on 3 level models

http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html#hiernormal

and her is some material on samp;ing weights in the MLwin software that I use

http://www.bristol.ac.uk/cmm/software/support/support-faqs/weights.html

I'd like to thank Kelvyn and Mervyn for their helpful / valuable comments and it helped me a lot to understand better on complex sample design and how to take it into account in our analysis. Thanks!

I am only a student and thus may not have great insight in methods like other contributors here but, i would strongly advise to look through

"Applied Survey Data Analysis, Second Edition Steven G. Heeringa, Brady T. West, Patricia A. Berglund" Examples of similar problems are shown with interpretation of results and assessment of the standard errors. all you need to do is identify the design variables ( PSUs, Final weights, stratum variable) these are always provided by the data provider if you read the documentation. once you declare these, the computational models are the similar and depending on the software of choice, you will obtain design adjusted standard errors with adjusted degrees of freedom.