I recently attended the Gallup Summit. During discussions with the survey and analytical folks there, I asked about the validity and other problems that arise as poll response rates fall. Gallup is averaging 10-15% response currently. This is compared to much higher response rates as recently as the mid 90's.
We discussed some of the issues outlined in the recent Pew release (link below).
While I understand the technical 'fixes', I'm left wondering about the respondents. Specifically, I wonder about systemic differences in the respondent pool and, therefore, their representativeness with respect to the population of interest.
The Pew findings illustrate, and to some extent validate, my concerns, pointing out that US minorities appear to be under-represented. My concern is broader.
As I understand it, non-probability sampling rely on the researcher's subjective judgement. While I appreciate, and have employed, this method in qualitative research, I've always understood it to be purposeful.
The quantitative application of what appears to me to amount to a self-selection process is, I believe, problematic when the findings are being used to generalize to a population without knowing if the respondents are representative of that population.
I'd appreciate and value the opinions and experience of my colleagues in helping me understand this issue.
http://www.pewresearch.org/fact-tank/2016/05/02/q-a-online-nonprobability-polls/
I don't agree that non-probability sample are subjective, because the people who promote them usually go to great lengths to "re-weight" them back to something like the known population proportions. How well this works is, however, open to further investigation.
In particular, the Pew report that you linked to says that some of the commercial, non-probability data sources they examined had substantial mis-estimation in minority sub-groups. This could easily occur if one of those panels had only a few such minority members, but weighted them heavily to make up for that under-representation. This is particularly problematic when the generation of the panel depends on people who use computers and email, which would under-represent low-income groups.
Interesting, Pew's own panel (originally recruited by random-digit dialing with additional mail outs) did fine on representing demographics, but also hd an "overrepresentation of civically engaged adults." The link did not report the response rate on the original Pew survey, but I have to wonder about the representativeness of people who would both pick up the phone and complete the survey (let alone the fact that it would not include cell phones, which are exempt from random digit dialing).
Did the sessions that you attended include any discussion of so-called "mixed mode" sampling where non-random panels are combined with random digit dialing and other methods?
David,
I expressed myself poorly. I meant to say that non-probability samples in qualitative research are chosen by the researcher and, in my view, that selection process is subjective.
I was trying to contrast that process with the quantitative application I saw at Gallup and as reported in the Pew paper.
As someone who does qualitative research, I agree that most of the sampling there is non-probablity -- otherwise knowns as "purposive sampling." As for the subjectivity factor, I say that the goal should be to avoid known sources of bias rather than to achieve generalizability, In any event, a random sample of 20 or so informants isn't generalizable in any meaningful sense, so purposive selection would be much more reasonable.
William -
For response propensity/heterogeneity group weighting to help, I would expect that the very low response rates, 10 to 15%, would be rather problematic. If you are fortunate enough to have regressor data, if bias is controlled well enough by data groupings, and variances can be estimated, you might at least have some idea of accuracy, and, in fact, accuracy would be greatly improved. But in many situations, such regressor data may not be available.
Representativeness is a term with some history, and has a different meaning for different people. I'd say that as long as your sampling method and estimation methods are compatible for estimating missing data reasonably, then the "representativeness" of a sample is satisfied.
I know that Mike Brick has done good work on a variety of nonprobability sampling issues, though my experience is more restricted. I worked a great deal on prediction for skewed establishment survey continuous data, for official statistics, with emphasis on estimation of the variance of the prediction errors for totals. For a more general view from Mike and others, of various techniques, you might find parts of papers at the following three URLs to be of interest to you:
1) https://www.researchgate.net/publication/273561892_Summary_Report_of_the_AAPOR_Task_Force_on_Non-probability_Sampling
2) "Beyond traditional survey taking: adapting to a changing world
Explorations in Non-Probability Sampling Using the Web" -
Proceedings of Statistics Canada Symposium 2014. - J. Michael Brick
http://www.statcan.gc.ca/sites/default/files/media/14252-eng.pdf
3) "The Future of Survey Sampling" -
Article in Public Opinion Quarterly 75(5):872-888 · December 2011
Impact Factor: 2.25 · DOI: 10.2307/41345915
J. Michael Brick
https://www.researchgate.net/publication/261967521_The_Future_of_Survey_Sampling
Cheers - Jim
Article The Future of Survey Sampling
Article Summary Report of the AAPOR Task Force on Non-probability Sampling
- Badges
- Science topic
- Similar topics
- Publishing