Thanks for your answer! I actually ment something different when I said 'randomly matching multiple times'. Assume you want to match your case and control group with a 1:1 ratio. Then randomly machting could for example result in:

Subject 1 in the case group matched with subject 4 in control

Subject 2 machted with 13

3 machted with 1

and so on...

Then you could do all the statistical tests as planned.

If you repeated this whole procedure, then your randomized matching could look like:

1 matched with 5

2 matched with 15

3 matched with 1

and so on...

All the statistical tests would now give very different results! Hence, if the randomized matching procedure is done multiple times you could potentially 'misuse' it to optimize for a maximum number of significant results. This is what puzzels me.. I hope I made my point cearer this time?

Why would you match individual values? If age and gender do have some relevant impact on the response that would interfere with the experimental condition, that one would statistically adjust for their effect. Matching, as you said, sounds so medival...

I am matching participants not individual values

I don't understand. You do the procedure. Find your result. Why do you repeat it. If you mean are there papers that say that you shouldn't simply keep fishing for a result until you find what you want, Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called

psychological science. Perspectives on Psychological Science, 7 , 543–554. is a good paper.

I should add there are reasons to repeat the matching. By doing it multiple times it will give you an idea of the variability due to those random elements in whichever procedure you are using. Here is a paper talking about doing this: Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency

through a multiverse analysis. Perspectives on Psychological Science, 11 , 702–712.

But, as Jochen says, there are other approaches if you are using matching just as a post hoc way to account for some variables.

@Daniel Wright, you summarized it very well by this:

I don't understand. You do the procedure. Find your result. Why do you repeat it.

I could ask you: Why wouldn't you repeat it? I feel like no-one really looks into that, but it does change the result quite significantly. Who wants his/her results to be depending on a randomized procedure? To me it seems randomized age and gender matching protocols in case-control studies can be mishandled quite easily.

@Lukas, call it whatever you want - matching values or individuals. You match individuals that do have similar values in some cofactors (e.g. age and sex).

Daniels point is well taken (if I got that right): if there are several ways to match, it is a good idea to reapeat the matching in all the different ways (or, at least, in some random subset), to get an idea of how much any particular choice of matching could impact the conclusions. So the message is: if you can repeat matching (each time matching different individuals), then you should do so and evaluate the variance between the results you get. However, your point is that authors could do all these matchings but report only that one for which they happen to get the "best significances". And here Daniels post also provides an answer: if different ways of matching are possible, the variance introduced by the different matchings should be reported. If (multiple different ways of matching are possible and) it is not reported, that can be an indication that the authors try to cheat and that there conclusions may not that well be supported by the data as it may look.

Thanks Jochen, that makes sense!

1. Jochen, you got my point.

2. You explained it better than I did!