I would not pick the first 2 options as the difference in pretest could cause colinear with the IV (groups). it is difficult to identify whether the effect of interaction is caused by the different intervention(group) or the pre-test score.

I would recommend to pick option 3, while Stratified sampling could be applied.

Thanks dai, interesting. Yes indeed I would check for collinearity of the covariate, but previously I have found very little relationship between variation in pretest measures of the IV and systematic changes in the IV across groups (or indeed posttest iv).

To play devil's advocate, if you abandon random assignment at the end, and assign on pretest iv, surely you would be dealing with two different kinds of people (not equivalent on variables both measured and unmeasured. Surely there is less risk of that by sticking to a combination of random assignment and replication.

Is there some trendy trend away from random assignment in quantitative circles that I missed?

Lawrence Patihis

That is a good point on "different kinds of people".In my opinion, If you think your pretest measurement does have effect on the pre-score, that matters. normally, it would be a good practice to have done all pre-test before the random assignment for all subjects in the study design.

And let's see the definition of random sampling: Random sampling is a procedure for sampling from a population in which (a) the selection of a sample unit is based on chance and (b) every element of the population has a known, non-zero probability of being selected. from https://stattrek.com/statistics/dictionary.aspx?definition=random%20sampling

Personally, if the method doesn't violate the two principles, I would still consider that a valid method.

Good question. The p value assumes sometimes random allocation produces differences, but that doesn't really help you with the scientific circumstances you have. The attached might be of interest (it is under review). It compares some approaches like those you list for when non-random allocation is used. The problem is my main conclusion is the choice between 1 and 2 depends on assumptions about the relationship between the pre-score and allocation. For you this is random. Could be an interesting situation to explore with simulation methods.

Many thanks Daniel and Dai (just to be clear I don't have such data, but was wondering what I would do when it happens).

Daniel: would you personally recommend (in memory distortion research, lets say) always randomly assigning throughout the data collection of an experiment?

It makes it easier, but in general random assignment is not the most powerful if you have background information on can match on these. In memory distortion research, though, you don't usually have variables to match on (though say in developmental research folks often bin subjects by age). If you had what you thought was a reliable measure then you match on this and take it into account. A fairly common approach is match on things like time of day and day, but usually these aren't then used in the analyses. What are you thinking of?

That is so interesting that getting pretest info and then matching into groups is more powerful than randomization. In most work in my area, examining pretest measures and matching into groups would be just about impossible because of the gradual process of participants signing up and participating at different times over a period of 6 months.

Student/Gossett gives an extreme example in discussing the Lanarkshire milk study where they should have matched on genes (used twins) rather than random allocation (which didn't work anyway). That is a good paper, I used to have it on reading lists. In memory distortion studies that kind of matching is likely impractical. Matching though does (if done well) account for a bit of the variance.

Thanks Daniel, my understanding of why matching on some variables might be held in higher regard in some researchers minds is clarifying. We used matching on age and gender in a previous low-n study, and I understand completely why you would match on twins (genes). At the same time I understand with a bit more clarity why I was so adverse to matching on some individual differences measures that could result in groups containing completely different types of people. For we had a suggestion to match on pretest measures of people's evaluations of a someone in a close relationship: that could inadvertently assign different types of people to different groups (likely differing on third variables we don't measure or don't even think about). So matching on gender, age, genes, makes sense, but uneasy about matching on some other variables. The other suggestion was to check if random assignment had evenly distributed evaluations of close relationships near the end of data collection, and then purposely and manually assign individuals to groups to balance out the groups with 50 more manually assigned participants. I am so concerned about unknown confounds that could come from mixing random assignment and manual matching right at the end. Practicality is another reason in addition.

I actually have the exact same problem in my research. I only controlled the demographics variables between groups and they seemed indifferent. I thought that was ok and enough for random allocation. But when i analysed group differences in scale variables i found group difference in just one scale which is the core for my study. Unfortunately it was too late because i've already started the intervention. So, i think 3rd way is not suitable for my situation (I've no time and budget to get new-more participants for a group skills training.) But if i choose 1st way and take the T1 mean as covarite, can I say anything about timeXgroup effect? So how can I investigate the effect of intervention on DVs? And Dai, what could be IV collinearity in pretest-posttest design, i think i couldnt understand it.