Dear Marjan

I will try to sumarize what the colleagues have said about the requirement of a probability sample for doing inference.

In first place, if we are talking about statistical inference on a population by mean of an observational study, then it has to be based on a representative sample. The only unobjectionable method to produce a sample that almost always is representative is a probability sample, but may be examples of non probabilistic samples that were representatives.

For example, if in a opinion poll, I take the first phone of each page of the directory of a city, although this method is not probabilistic, it is possible that the sample was representative of the population of electors in the same city. However there were examples like this that with careful consideration showed some bias. Then if this is in the context of an observational study, we have not many alternatives to the probabilistic sample.

However in the context of experimental design, as Olga pointed, you are doing inference over a convenience sample. With the randomization of the treatments to the experimental units, you are generating a virtual population of all the allocations of the treatments to these units. The observed data are a sample of this virtual population and you may obtain evidences for the effects of each randomized factor on the units.

Let assume that we found that a treatment contributes to a better result on the treated units, then we may extrapolate this result to the population of all the units, that are like that in the experiment (like that in the convenience sample). This final inference is not a statistic inference, but an inference based on the expert knowledge, if the convenience sample is a representative sample this is more easy of justifying.

The best situation is when the experiment is done over a probabilistic sample of the units of the population on which we want to do the final inference. Then the statistic inference on the results obtained in the experiment may be applied directly to the objective population

Marjan, what do you exactly mean of non-probability sampling?? In most cases of sampling, your sample is a random anyway. For example, if you want to study some attributes in people, even your friends are a random sample, unless, the target variables has something to do with you.

A convenient snowball sampling may have some bias. For instance, if you take questionnaire from people you see in one day (e.g. Sunday) in a park, they may have a common purpose to go there ant it is a bias, unless you work on that purpose.

You mentioned subgroup in your question, this means you can increase randomness by applying a cluster or stratified sampling.

@Ehsan Khedive: I don't understand how anyone's friends would constitute a random sample in the sense "a small sample selected at random to be representative of a larger population". There would be an unavoidably bias there, I think (no matter what we're studying, be it political science, psychology, epidemiology, or linguistics).

I think the original question would be easier to answer if more context was provided. What's being studied, how, and for what purposes? Statistical methods are an important part of linguistic research methods, even if questions of sampling can be difficult there too. However, answers to those problems are best situated within a linguistic context and informed by knowledge of linguistics. My guess is that subject matter context would be important when dealing with those issues in other fields too...

Thank you so much for answer but I did not get my answer!

What's the goal of your research? Most, if not all, of the experimental research uses convenience sampling and yet we are perfectly comfortable with using inferential statistics to analyze experimental data. This is because the goal of the experimental research is to establish the effect of the treatment, not to describe the population. If the goal of the study is to describe the population (as it is with most surveys), then you should employ probability sampling methods.

In other words, there is no rule that states "you mustn't use inferential statistics with nonprobability sampling methods". The choice of statistical method is determined by the nature of your data (e.g., measurement scale, violations of normality, and so on), not by your sampling methods.

@Marjan J: As I tried to say in my post above: if you tell us more about your project and its aims, it will be much easier to answer your question. As I hinted at, and as Olga Lazareva expressed much more clearly than I did, the answer depends on a number of factors.

Example factors would be:

* Are you trying to draw inferences from a sample to a population?

* Is the study experimental or based on incidence or survey data?

* If you want to draw inferences / make predictions for a population, what is it a population of (school children, voters, light bulbs, batches of beer, speech communities, etc.)?

* What do you know already (from theory, models, previous research, etc.) about the relationship between the population, your variables, and the phenomenon you want to study?

All of these factors would, at least to some extent, play a role in deciding the choice of statistical method, as well as how far you can draw your conclusions.

I agree with Olga Lazareva that there are no absolute hard rules in this matter. However, I do not completely agree with the statement that the choice of statistical method isn't determined by the sampling method. Some statistical methods are better suited for non-independent observations, to the point of allowing some relationships in the data to be built directly into the model (such as with multi-level methods). I still agree with Olga that things like measurement scale, assumptions about data distributions, etc. are pretty fundamental, though.

@Gard: You are correct, of course, that some methods are better than others for non-independent observations. But this issue, in my view, is somewhat orthogonal to sampling: I can use convenience sampling and between-subject design, and voila! independent observations only. The nature of design (at least in the case of experimental research) is more important than sampling methods when it comes to selecting statistics. The nature of sampling method becomes more important when it comes to interpreting statistics: If we have a convenience sample at a lily-white undergraduate college, we better be careful about making claims extending to population at large.

@Olga: I see your point. I think, on reflection, that this at least partially comes down to the type of research. I mainly do corpus linguistics, i.e. non-experimental research with incidence data. In the absence of control over an experimental situation, I typically find that the two issues are very much intertwined. Especially in historical linguistic corpora, strong biases are often present in the sampling, so this is usually something that needs to be constantly taken into account. As you say, in an experimental situation things are somewhat different.

It seems to me that we very much agree about the core of the question, which is that the wider research context is crucial to take into account, and that it will inevitably influence how we think about sampling, choice of statistical methods, and interpretation of the results.

Dear Marjan

I will try to sumarize what the colleagues have said about the requirement of a probability sample for doing inference.

In first place, if we are talking about statistical inference on a population by mean of an observational study, then it has to be based on a representative sample. The only unobjectionable method to produce a sample that almost always is representative is a probability sample, but may be examples of non probabilistic samples that were representatives.

For example, if in a opinion poll, I take the first phone of each page of the directory of a city, although this method is not probabilistic, it is possible that the sample was representative of the population of electors in the same city. However there were examples like this that with careful consideration showed some bias. Then if this is in the context of an observational study, we have not many alternatives to the probabilistic sample.

However in the context of experimental design, as Olga pointed, you are doing inference over a convenience sample. With the randomization of the treatments to the experimental units, you are generating a virtual population of all the allocations of the treatments to these units. The observed data are a sample of this virtual population and you may obtain evidences for the effects of each randomized factor on the units.

Let assume that we found that a treatment contributes to a better result on the treated units, then we may extrapolate this result to the population of all the units, that are like that in the experiment (like that in the convenience sample). This final inference is not a statistic inference, but an inference based on the expert knowledge, if the convenience sample is a representative sample this is more easy of justifying.

The best situation is when the experiment is done over a probabilistic sample of the units of the population on which we want to do the final inference. Then the statistic inference on the results obtained in the experiment may be applied directly to the objective population

Marjan - My specialty - if you use regressor data. And yes, lots of inferential statistics are possible. "Prediction" is the term for the estimation. (It sounds like forecasting, but it isn't.) You could start with "Efficacy of Quasi-Cutoff Sampling and Model-Based Estimation For Establishment Surveys - and Related Considerations," in my area, and the last five pages of "Cutoff Sampling and Inference." Also "Properties of Weighted Least Squares Regression for Cutoff Sampling in Establishment Surveys." There's a great deal more. - Thanks - Jim

Regarding not knowing the "total population," I'm not sure what you mean by that. if you have good regressor data, and the classical ratio estimator, you may be able to get around that, if you have regressor data for the entire population. But it sounds like we need to know more about your application. - If you mean you need network sampling, that may be feasible. That sounds like maybe an area for a biostatistician.

:-)....u r right Marjan J........... ....inferential statistics means the test like chi-square, ANOVA, MANOVA, t test, regression, factor analysis...........if your sample is a non probability sample then you cant use some of the inferential statics like t test, regression, ANOVA, MANOVA... but you can use friedman ANOVA, K-S test, Kruskel Wallis test......

Dear Arvind

You did a serious mis-presentation of the matter. You are confunding parametric methods, with methods applicables to probability samples, and non parametric methods, with non probabilistic samples.

Both the parametric methods and the non parametric methods, are fundamented in a probabilistic sample or a randomized experiment. The random selection of the sample or of random assignement of the treatments to the experimental units determines the validity of the parametric or non-parametric models which may be subsequently applied to make decisions about the population under study.

When we have not a probabilistic sample, sometimes we can obtain a representative sample without using a probabilistic sample, because the nature "randomized" the units for us. In the example of the directory (above), the order of the persons listed in it, has no relationship with the opinion poll and we can consider it as a "random population". Then in this situation we may use parametric or not parametric methods on accord of the type of variables considered.

The parametric methods are valid when the variable is normal or when the conditions behind the Central Limit Theorem were met. The non parametric methods may be used besides in situations where the parametric are not appropiate (non normal, ordinal variables, with heterogeneus variance...).

It was R.A. Fisher who introduced randomization into experimental design and brought statistics into the realm of the practical. Note that randomization is mathematically defined and is not happenstance or convenience. To quote R.A. Fisher, "A real experimenter, in fact, so far from being willing to introduce an element of chance into the formation of his scientific conclusions, has been steadily exerting himself, in the planning of his experiments, and in their execution, to decrease or to eliminate [by randomization]* all the causes of fortuitous variation which might obscure the evidence.”

Statistical Methods and Scientific Inference, Edinburgh: Oliver and Boyd, 1956, p.97; *3rd edition, 1973.

Thank you Charles, for the quote of R.A. Fisher!

Like most to explained, IS NOT CORRECT apply inferential statistics in non-random samples, but mathematically you can use inferential statistics in any context while having quantitative data. BUT THAT IS NOT THE PROBLEM, inferential statistics used in non-probability sampling. can not say that the differences or associations are attributable to the variables involved, but may be a result of the sample itself.

Marjan J. Some provisional ideas to 1) "We can only use descriptive statistics for non-probability sampling". Histograms are graphs that include frequences -akin to probabilities-. 2) "Do you know any thesis or article which has used non probability sampling, but still used inferential statistics?": I read from philosopher Enrique Dussel that Aristotle collected lots of caracol species from greek islands just to infer: "they share the shape of spirals". 3. "I do not know the total population and even the number of them in every subgroup if I divide the total population. If I use convenient and snow ball sampling, then can I use inferential statistics?": This is frequent in sampling related to wide populations: we only gather data from limited contexts which may be considered a basic premise of the research and conclusions only apply to it; but if you get N values of your variable, eventhough we ignore frequences, it is normally applied the Laplace criterion, which assumes that each value is a "mean value" of a sector with frequence "1/N", so you accept that all your sectors have equal chance, but variable media may be different. Therefore we estimate the U media of the sample as a weighted average in some cases, and as a simple average in other ones. This is just to compensate somewhat our ignorance in this "non controlled" samples.

Another hard point is how to know on time that our dataset is "representative" of a much wider context. As you, I also need help about this point. Regards, emilio

In survey statistics, random sampling can often go awry, so it is much better to have model-assisted, design-based sampling. That is, auxiliary (really, regressor data) help with the problem of being "representative." Mike Brick gave a presentation not long ago for the Washington Statistical Society (chapter of ASA) in which he noted that there are many kinds of non-probability sampling, some not as good as others. What really matters is whether or not you have regressor data. - But this may not help Marjan J, as it seems that the application that started this thread may not lend itself to this approach. I would like to hear more about that application, or are we discussing various applications?

Thank you all. My PhD thesis is survey. I used convenient and snowball because I do not have the list of people to choose from them randomly! I participate in the events related and give out the questionnaire to almost all of them of my target population who accept to fill out. for example, if I go 10 events, I will almost give all of them because fining my target population is not easy. I decide to use Amos to confirm the model of study and I am confused how justify this study in order to be able use inferential statistic .

@ James Knaub The application is in social science , wildlife management and conservation issue. The base of model is theory of panned behaviour.

@James Knaub: Do you have the text:Washington Statistical Society (chapter of ASA) in which he noted that there are many kinds of non-probability sampling,

What should I do with my data for analysis!

@ Marjan J: what you do with your data for your analysis is a different question. In short, it depends on what you want to investigate, and what types of data you've got.

What kinds of data do you expect to get back from your survey? Nominal data ("what is the name of your favorite chocolate bar")? Ordinal data (Likert-scale rating responses)? Ratio data ("respondent's age", "number of hours spent watching TV previous week")? All of the above?

Is the study a pilot study in a previously less-researched area? Is the study intended to be mainly descriptive? Does it aim to replicate previous research in some way? Or is the goal to test some substantial theoretical claim?

Your answers to these questions would (ideally) help to guide you in the choice(s) of statistical method(s), and how to proceed in your analysis. If you're wondering about the choice of statistical methods, you can check out my paper "Student attitudes toward teaching English with Technology", which is available from my profile. The study design itself is a bit simplistic, but it should give you a good idea of the different kinds of statistical methods you can use for survey data.

@ Gard I am confused because I should a analysis data by inferential statistic to answer the hypothesis of my thesis. But the problem is that if the method is non probable , I can not apply inferential statistic in my survey.

@Marjan,

as the discussion above makes clear, the statistical methods don't care about whether or not your data arise from random sampling.

If you have a well-designed study with lots of respondents, collected with a validated measurement instrument, through random sampling, then it's easier to say something about the population you're sampling from. If this is not the case, then it's harder (perhaps impossible) to say something about the population as a whole.

However, that doesn't mean you can't use statistical methods to describe patterns and differences within your sample. It's about the interpretation, not the statistical methods per se.

I don't know why you "should" use inferential statistics, if it doesn't make sense given your data collection method and/or hypothesis (only you can answer that question). You can anyway employ summary statistics and exploratory methods, such as plots, dimensionality reduction plots (correspondence analysis, multi-dimensional scaling...), or tree-based clustering methods. There are many possibilities out there.

In answer to what you addressed to me: This does sound like a convenience sample. As for a snowball sample, isn't that a network sample where you ask people to respond and for them to ask other people they know to respond? I think there has been a lot done on that, but not something I know. I will get in touch with Mike Brick and see if he has slides from that seminar that you might find helpful, but it could take some time. He is not on ResearchGate. -- As for inference, I know two ways to approach that. There is the principle of randomization, and the principle of conditionality. They are, respectively, behind design-based and model-based sampling and estimation. (There is also model-assisted design-based estimation.) Ken Brewer did a lot to put them together. What I think you have here is a chance to use modeling (regression), not to estimate totals from finite populations, but just to describe relationships between variables in your study. Best if you can stratify your data into groups, and within those groups, you may discover relationships between your questions that may further your analyses. You have to be careful though. For example, I got too much sun when I was young, and I get a lot of pesky basal skin cancers. A doctor once said people with dark hair usually don't have that problem. He was not really looking at it the right way. People with darker hair usually have darker skin. Dark skin is protection from skin cancer. Dark hair does not help at all! So if you find relationships in your data, you have to ask, "Why?" But if you regress one data element (question) against another, and get regression coefficients with relatively low standard errors, you may discover something interesting. For example, in a simple regression through the origin, one regressor, you could get a y = bx + e (where e is heteroscedastic, say a model-based classical ratio estimator) and get standard error of b, and be able to relate x to y nicely if that standard error of b is small compared to b itself. This would likely be within a given stratum (category).

Gard said something about patterns but not the whole population. That sounds reasonable.

Gard,

You stated " the statistical methods don't care about whether or not your data arise from random sampling." There are simple summary statistics that don't care about either random sampling or randomization (means, percentiles, standard deviations, frequencies,....). However, anything called inferential statistics will have a theoretical basis in random sampling (including complex forms of random sampling), randomization, or both. I recommend that you look at the formal publications of the methods and the stated assumptions under which those methods are developed.

Chuck

Please Marjan,,, can you explain the Thematic your survey. is an elaboration itself. the scale you use. WICH is the question, that motivates the use of the survey?

As promised, resource obtained from Mike Brick:

http://www.aapor.org/AM/Template.cfm?Section=Reports1&Template=/CM/ContentDisplay.cfm&ContentID=5963

@Charles: thanks for the comment, guess I was shooting from the hip in that passage you're quoting.

Inferential statistics in its classical Fisherian form is of course heavily based on assumptions of sampling and randomization, as you point out. The point that I was rather clumsily trying to make was that (some?) statistical methods can be employed to describe or model a sample, even when some of the formal assumptions are violated thus prohibiting making generalizations to a full population.

The position I describe here is perhaps a bit of a heresy viewed in light of Fisher's classical paradigm, but in practice I see it used a lot in publications. In my own field, linguistics, random sampling is usually not feasible, even when doing experiments: those undergraduate students who volunteer for, say, a reaction-time experiment can't be considered a random sample in any way whatsoever...

In practice, I often find myself working with "found data" of some sorts (be that Old English texts that happened to survive, or students who volunteer for an experiment). Since the goal in quantitative linguistics studies tends to be description or modeling of some sort of behavior, reaction or judgment, there are rarely any "treatments" to which participants can be meaningfully randomly allocated. Instead, the aim is often to tease out a correct model structure incorporating several small to medium sized effects (often in competition with each other, or with various interactions). I guess this situation has parallels in the social sciences...

I don't want to sound like I take the question of sampling lightly. I take it very seriously, and I've wrestled with the topic in some of my publications, precisely because it's hard to fulfill all those classical assumptions in a field like linguistics. However, I find that even when the random sampling assumption is not valid, some methods are still useful for working out the internal relationship between variables within a sample. There is of course a risk of false positives in such an approach, but I think that this can be mitigated to some extent by gently shifting emphasis away from statistically significant p-values, towards effect sizes and model fit.

Marjan,

If you plan on using Amos, then it sounds you want to build a complex Structural Equations Model (a form of regression, for those who don't know SEM).

The primary use for such an approach is to determine which effects do and do not belong in the model, and that is a question of statistical inference (i.e., effects that are not significant are dropped from the model). So, if you are going to be following the traditional assumptions of inference, the deficiencies in your sample would indeed be a problem.

But I personally think you could justify a "descriptive and exploratory" approach to your modeling exercise. In particular, that would involve using conventional statistical inference levels as a decision making tool for building the model, even though you have to admit that the results can not be generalized to any well-specified population.

Given your sampling strategy, I would guess that you are working in a field where the difficult of data collection makes it hard to produce empirical results. That always produces a dilemma about either giving up on doing statistical work at all, or accepting major limitations on what you can say about the data you do gather. In that case, what probably matters most is the judgments of your "peer reviewers" -- will they be pleased to get the best data possible under the circumstances, or will they reject the data out of hand if doesn't live up to standards that are virtually impossible to meet in the real world. So, you may be trying to answer a question that is as much about the values of your colleagues as it is about the laws of statistics.

Essentially, if you do it (convenience sample and inferential statistics) your results are problematic.  The research design does play a part of course when drawing inferences but stats are stats and if you want to use them in an exploratory manner nothing is stopping you.......but you need to justify their use in this way... fundamentally it probably won't fly with your peers.  However, clinical samples often have purposive samples and use inferential stats!!! You should be cautious however, as even in the 'real world' ...limitations should be noted...and you can't generalise. I have used 'clinical samples' but I don't find myself comfortable with it!!! The answer - re-design your research

can a large sample size be used to overcome the limitations of non probability sampling technique for generalization purposes

Stratify

Dear Marjan

Ya, it would not be generalise for the whole population i.e. precisely you can not make the findings for whole population under study. 

However, if your population is small and you can jusitfy theoretically, atleast some relevance can be obtained from your data.

Otherwise, you can use this to define your population and henceforth you can draw a random sample. True for some cases.

See my detailed response to https://www.researchgate.net/post/Does_large_sample_size_overcome_the_limitations_of_non_probability_sampling_techniques_for_generalization_purposes

I have the same doubt with my thesis, how did you solve it? Thank you

I am having the same problem with this too.

Hello

From the research methodology, the sampling process is not only the sample size, but how was it selected? If the process is random, then it is very likely that you can make some inferences towards the population. But if the selection was based on criteria without considering the dynamics of the population, it is very little possible that they validate the inferential process towards the population.

Now, the discussion is in, does the research want the findings to be occupied in similar environments? If your answer is yes ... then sample correctly, if not, explain from the case. DO NOT generalize.

hi carlos,

Purposive sampling can be used in your case as well as you can use inverse sampling too when population size is unknown. Only problem in purposive sampling is that there is no probability attached with the units selected hence drawing conclusion will become little difficult than if you go with inverse sampling with post stratification.

Hi Kaustav--

You're right, but Mosa and Maria have the problem

In general, these are all issues that need to be mentioned under the heading of "limitations" in either your Discussion or Conclusions section. If the issues you are facing would be the same for almost anyone trying to do research on your topic or with your population, then you can expect your reviewers to understand that this is an inherent limitation, rather than a specific problem with your study.

Dear Colleagues

The problem with non probability sampling is that they are more susceptible to biased results on the estimation of the parameters of a population, and on the estimation of their precision. This sampling methods may produce good or bad estimations but their reliability is far more subject to controversies. If hypothesis testing were also done, their conclusions could be seriously distorted. Then any time that it is possible to use a probability sampling design, it is best to use it, although it may become a little more onerous. The very process to build a suitable sampling frame may help to improbe the quality of all the study. It is also convenient to use complementary model based methods over the probability sample to improbe our inference taking notice of all the available related information.

For an explanation of the model-based approach, see "The model based (prediction) approach to finite population sampling theory," Institute of Mathematical Statistics Lecture Notes - Monograph Series, Volume 17, 1992, 225-240, Richard M. Royall,

 https://www.researchgate.net/publication/254206607_The_model_based_prediction_approach_to_finite_population_sampling_theory/. 

The paper is available under Project Euclid, open access:

 https://projecteuclid.org/euclid.lnms/1215458849.     

You might also consider the following: 

An Introduction to Model-Based Survey Sampling with Applications, 2012, Ray Chambers and Robert Clark,  Oxford Statistical Science Series

But it does not sound like either the probability-of-selection-based (design-based) approach, or the model-based approach, or a combination will work for the question asked. I have seen work before on what it takes to make snowball sampling work, but I don't recall where.