Hello Ita,

Most folks would consider results from randomized control trials as superior evidence for estimating the magnitude of an effect due to a target IV compared to results from non-randomized studies. It is certainly your option to include RCT as a requisite attribute for studies to be included in your synthesis. If you decide to include both types, then the usual practice (in my experience) is to report results separately for randomized vs. nonrandomized studies, to see whether that study attribute made a difference in the estimate of effect size. If there is no difference (beyond a trivial amount), it make sense to combine the two into an overall estimate.

However, randomization does not guarantee that any single study will have equal, exchangeable, groups a priori. What it does offer is: (a) more confidence that the results are not due to opportunistic sample creation/selection; and (b) over an infinite number of trials, there will be zero aggregate bias (on average) between treatment and control condition cases on relevant and nuisance variables.

Good luck with your work.

Dear Ita,

From a technical standpoint, randomization is a requirement for the probabilistic interpretation of the regression model, not the fitting of the model. The purpose of the randomization is to reduce the influence or biases of the researcher in selecting a sample so the sample will fit the probabilistic model, and hopefully, resemble the population. While using nonrandom data does violate an assumption for a pure probabilistic interpretation of the results, it is seldom achieved completely even in published studies. When that is case, the nonrandomness of the data should be included as one of the limitations of the study so consumers of the research can make their own judgments about the fidelity of your results.

Good luck,

Jim

By the way, "balanced sampling" is the model-based 'answer' to the benefit of randomization. In the simplest case, if there is one predictor, x, you take a sample where the mean of the x-values associated with the y-values of the sample is about equal to the mean of all x-values in the population.

One could balance on predicted-y, I'd say.

Balanced sampling basically nearly guarantees what random sampling only hopes to obtain in this discussion.

I don’t really understand what you’re saying. Could you expand please?

Dear Ita

you can check impact on predictors individually and accordingly decide on choice and randomization.

Ita - Just saw your note to me:

When randomization is used to estimate totals and means by use of a "design-based" sample, to weight members of a sample to represent a population, then that is a very different use of randomization than to only make your sample likely distributed something like the population. But when you are using regression, all you might use randomization to do is hopefully have a sample distributed similarly to the way the population is distributed. People might often think of that as a "representative" sample. But when using regression to "predict" for missing members of the population, here weight gain, i would expect that you can do better than randomization for finding a "representative sample."

This is not exactly my area. I have worked a great deal with finite populations just to predict their totals, especially from establishment quasi-cutoff (multiple attribute) samples, to minimize uncertainty considering bias and variance, rather than using the probability-of-selection-based approach, but it seems to me that here you are interested in the concept of a sample which "represents" the population. When randomization is just used for that purpose, and not in the estimation process as in design-based sampling, then relying on randomization to obtain that goal of "representativeness" is particularly haphazard. There is always the possibility of an unfortunate "draw" of the sample, and here that risk seems particularly unnecessary. I would expect that you want to sample enough in all parts of the population, not too much nor too little in any one part of the population. So balanced sampling can do that.

A simple example of that concept would be your gender question. I think you wanted to know what happens "...if males vs. females are not evenly present within the sample?" Well that could be a problem, so instead of taking a chance that you could have all females or all males in the sample, why not sample them separately? This not only hints at the need for stratification, which is used to get better overall results, but also the idea of separate subpopulations, to make certain that you get good results for each group. In regression you could use a categorical value.

Anyway, I'm just saying that if you are using regression, and you only use randomization to try to get enough of each part of the population, you should use your knowledge of the population to be certain that you do just that.

This also leads us to the idea that your model may not apply to the entire population or subpopulation to which you wish it to apply. Maybe a different subpopulation might require a different set of predictors, or at the least, substantially different coefficients.

If anyone sees something I'm missing for the application at hand, I'd like to know. Thanks.

Cheers - Jim

That makes sense. Thank you for your valuable help!