Promote increased replication. Every experiment is really two experiments. The first is the planned experiment that is conducted. We hope it has sufficient replication. However, I could go back and take another sample from the population that you sampled and I would get a different answer (at least quantitatively). So the experiment is one replicate of all possible experiments that could have happened. If you increased replication there would be room to do other types of analysis that could start to address this concern.

The place to start this would be an introductory statistics course so that more people consider experimenting on broader issues. However, this might be a problem for people that struggle with statistics.

I got to the second controversy in the Brewer article. Point 1, gave me trouble "Because randomization had not been used, the investigators had not been able to invoke the Central Limit Theorem. Consequently they had been unable to use the normality of the estimates " I think this statement is nonsense. The distribution of the individual sample will converge to the underlying distribution of the population being sampled. If I sample selectively, then the techniques used to choose my sample points also define my population. So random sampling doesn't seem to enter into this On the other hand, my experiment in one of many possible similar experiments unless the technique for sampling is somehow deterministic. This is not allowed: I will sample the first person named Sally who walks through the hospital door at 8:00 on Tuesday, and I know that the department head always arrives at 8:00 on Tuesday and her name is Sally (yes she could have a cold on my sampling day, or get stuck in traffic, thereby resulting in a small probability that I would get someone else). However, as long as there is some random process that is sampled, then the distribution of all possible experiments that I could do will be Gaussian by the CLT. Randomization, in terms of sampling design, doesn't enter into this. If I have an available population of 5000 people, and I assign each of them a number, then have a random number generator spit out 50 numbers and I use those people the CLT will apply to my sample. If I select every 100th person the CLT will still apply unless someone is making sure that all the people go through the door in exactly the same order every time I run the experiment no matter which door I use. Randomization may help insure that the sample is not deterministic, but it is not a requirement for the CLT to function.

Replication is an important idea for experiments.  Thanks.

When considering the CLT, randomization is used in a role of representativeness.  It may be that for many experiments this naturally occurs.  Consider the explanation here: https://onlinecourses.science.psu.edu/stat414/book/export/html/133

It is important to consider the meaning of a random variable. 

Design matters, for example, because you cannot take all data from one stratum and always assume it well represents all strata. 

Granted, however, the notion of total survey error that this question wishes to explore is How much difference can each violation of assumption or other source of error make, from an overall point-of-view, i.e., 'all things considered'? 

Cheers.

Ah, but part of the problem is that random does not equate to representative. The other problem is with random. If I sample a population that is 50-50 male:female and I withdraw 50 individuals then I have a 0.6% chance that I will get a sample that is skewed 15 of one gender or fewer (35 or more of the other). While random, would any of these be acceptable? If the random sample that is all female is discarded, is the sampling method still random?

Thank you James for the very nice link.

In a broad sense the question is unanswerable. The answer will change depending on the questions asked, the field of study, and the methods of analysis. What is the quantitative definition for failure? It is not that we should not try, but we probably cannot arrive at a single metric that we would all agree on for making such a determination.

Note to all: Stratification and poststratification are very useful tools, by the way.  Even for strictly model- (regression/'prediction')based sampling and estimation, stratification is hugely important.  (Also, starting with a paper I published in 1999, I started referring to "estimation groups" [EGs] as any subset of a population for which a single model works well.  These EGs may overlap "publication groups" [PGs] such that one might even "borrow strength" in small area estimation, to allow publication at an aggregate level for "small areas.") 

Stratification, though very important, is only one consideration when trying to reduce Total Survey Error (TSE). 

What else can be done?  There are papers given for TSE, and an international conference I was never able to attend.  Has anyone reading this attended that or a similar conference?  Published any works in TSE?  Are there any journals which have published special issues on this? 

There are many considerations with regard to error, besides generating standard error estimates, but often even that is not done.  Conferences which encourage the study of TSE seem rare.  What can be done?  Perhaps more general conferences might sponsor sessions in this area? 

...  

I found this on the internet, and I believe Paul Biemer, noted there, is rather well known in this area: 

 https://projects.isr.umich.edu/jpsm/html_content.cfm?CourseID=050415

.....  

.....  

Suggestions? 

I would like to see suggestions and try NOT to interject my own opinion or comments any further. 

Any suggestions and/or experiences you might want to contribute are welcome.  Thank you all.  Know that I appreciate your contributions and others will also. 

....

....

One last thing I should note is that for design-based, i.e. probability-of-selection-based, sampling and estimation, the randomization assumed in deriving the estimators is in the selection process, but for regression/'prediction' model-based estimators, the randomness is found in the estimated residuals.  That is why a standard error is simply estimated from the data in design-based sampling, but in the model-based case, the estimated variance of the prediction error is estimated from the estimated residuals. 

These design-based and model-based methods for finite population sampling are sometimes used together - e.g. see:

Särndal, CE, Swensson, B. and Wretman, J. (1992), Model Assisted Survey Sampling, Springer-Verlang, and      

Brewer, KRW (2002), Combined survey sampling inference: Weighing Basu's elephants, Arnold: London and Oxford University Press.  

More on prediction is found, e.g. in Finite Population Sampling and Inference: A Prediction Approach,  2000,

Richard Valliant, Alan H. Dorfman, Richard M. Royall,

Wiley Series in Probability and Statistics. 

Total Survey Error refers to the accumulation of all errors that arises in

the design, collection, processing, and/or the analysis of survey data. Survey errors negatively affects the results of the survey and can be costly. I believe no way to eliminate total survey errors, but we can reduce them through many ways. 

I usually use non-response bias analysis across time in my surveys. Non-response bias analysis has to be done to compare between respondents and non-respondents. 

Dear All

Total survey error is a concept that purports to describe statistical properties of survey estimates, incorporating a variety of error sources and types:

· Non-sampling error

· Non-response error

· Response error

· Processing error

· Missing answers

· Sampling error

The attach paper may is useful for this problem,

https://watermark.silverchair.com/nfq065.pdf?token

Regards,

Zuhair