In Applied Survey Sampling, by Blair and Blair, Sage Publications, 2014, on page 175, they note the common use of "credibility intervals," by researchers using nonprobability samples and Bayesian modeling. They note that the American Association for Public Opinion Research (AAPOR) issued a caution to people using these credibility intervals, as not being something the public should rely on "in the same way" as a margin of sampling error. Attached is the AAPOR statement from 2012 in which they caution heavily regarding the use of such nonprobability opinion polls, as the Bayesian models have assumptions which will be of varying quality. However, they also state that "...even the best design [probability sample] cannot compensate for serious nonparticipation by the public." Thus, for much of the data in a probability sample, we have to use models or some method to estimate for missing data, and when the nonresponse rate is high, do we really actually have a valid probability sample anymore?
Thus the emphasis would be on total survey error. There is sampling error, and then there is nonsampling error. We have nonresponse and that can make reliance on a model better overall.
If that is the case, then why do many survey statisticians insist on probability samples for highly skewed establishment surveys with continuous data, when excellent regressor data are available? Often sampling the largest few establishments will provide very high 'coverage' for any given important variable of interest. That is, most of the estimated total would already be observed. The remainder might be considered as if it were missing data from a census. But these missing data, if generally close to the origin in a scatterplot of y vs regressor x, should have little prediction error considering the heteroscedastic nature of such data. With the relatively high measurement error often experienced with small establishments, long experience with energy data has shown one will often predict values for y for small x more accurately than one could observe such small y's. Further, this is done using the econometric concept of a "variance of a prediction error," and Bayesian model assumptions are not introduced.
It is important not to lump nonprobability sampling having good regressor data with other nonprobability sampling. For official statistics, an agency will often collect census surveys, and have more frequently collected samples of the same variables of interest (the same attributes), or a subset of them. Often the best regressor data for the samples are from such a census.
Finally, many years of use in publishing official statistics on energy data have shown this methodology - far less radical than the use of "credibility intervals" for polling - to have performed very well for establishment surveys. It does not appear reasonable to argue with such massive and long-term success. The second link attached is a short paper on the use of this methodology. The third is a study of the variance and bias, and the fourth shows a simple example, using real data, as to how effective a quasi-cutoff sample with a model-based classical ratio estimator can be.
Since the 1940s, for good reason in many cases, probability sampling has been the "gold standard" for most sample surveys. However, models are used heavily in other areas of statistics, and even for survey statistics "model-assisted design-based" methods have unquestionably often greatly improved probability sample results. But strictly model-based sampling and estimation does have a niche in establishment surveys, though it has met with resistance. One should not dismiss this without trying it. It especially seems rather odd if anyone were to consider Bayesian "credibility intervals" for election polls, but not quasi-cutoff sampling with model-based estimation, as defined by going through the second link below.
Your comments?
http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/DetailedAAPORstatementoncredibilityintervals.pdf
Article Efficacy of Quasi-Cutoff Sampling and Model-Based Estimation...
Article Using Prediction-Oriented Software for Survey Estimation - P...
Conference Paper Projected Variance for the Model-based Classical Ratio Estim...
Consider continuous data for a finite population. A random sample can easily be drawn which is substantially "unrepresentative" of the population, especially when there may be a few members of that finite population which may be unknown to be outsized. That is why model-assisted design-based sampling can be so useful. Sometimes the model may be more important than randomization. For many highly skewed establishment surveys, a ratio (regression through the origin) model, applied by appropriate data groupings/strata, along with relatively high 'coverage,' are what is important. For small size x, predicted y may be more accurate than if observed, due to measurement error. In Applied Survey Sampling, by E. Blair and J. Blair, Sage, 2014, on page 170, they note sometimes higher nonresponse for such cases, as smaller entities may have less facility for responding. When they do respond, much work has indicated that data quality may be especially low. Thus, cutoff and quasi-cutoff sampling (defined in the paper found through the link below) may provide better inferences in such cases, and far, far more efficiently.
The paper at the link attached briefly addresses: (1) How do we know this is performing well? and (2) What if the smallest members of the population appear to behave differently? This review article contains a summary of conclusions from experimental findings, and explanations with numerous references. A huge amount of official US energy statistics data have been published for about a quarter of a century, and borne-out over time.
So from a total survey error (TSE), best inference point-of-view [though it is best to be able to combine randomization and regression/ratio modeling, a la Ken Brewer], for highly skewed establishment surveys, relying on models applied by groups, that can be identified using scatterplots, is preferable to randomization. This may seem counterintuitive without proper investigation, but has long been proven to be true with much data published at the US Energy Information Administration. (Probability proportionate to size sampling and stratified random sampling with a certainty stratum are other possibilities, but various issues have demonstrably shown them to be far more problematic for highly skewed energy industry survey data, and likely other establishment surveys.)
Arguments against nonprobability election polling appear more convincing, even with elaborate Bayesian modeling. There appears to be far too much chance of impact from unknown sources/variables. However, perhaps someone has shown under what conditions this performs consistently well, and I am unaware?
Why would it appear easier to convince more people that nonprobability polling samples (with Bayesian modeling) may be useful, than nonprobability/quasi-cutoff establishment survey sampling (with a ratio estimator)?
Is this a problem that could better be approached as a psychological challenge when people will not examine the evidence, and will not budge from preconceived notions? How should this be addressed?
Article Efficacy of Quasi-Cutoff Sampling and Model-Based Estimation...
- Badges
- Science topic
- Similar topics
- Psychology