Therapists is a random factor here. But if you run a mixed-effects model, what you will get is a term showing how much variance there is between therapists; it's not clear how interpretable that will be.

In general, it will help to express your hypotheses more specifically. If your hypothesis is just that different therapists are associated with different treatment effects, that hypothesis sounds kind of empty to me (admittedly not an expert in your field) because the alternative ["all therapists are the same"] is not plausible. Things in the world vary, so I'm sure even without running statistics anyone would already expect that different therapists will be associated with different values on the dependent variable. Is there any more specific hypothesis you have (e.g., are there certain traits in therapists that you think will be associated with better outcomes)?

I agree with Stephen Politzer-Ahles . If you could categorize the 60 therapists into a few categories (say, less than 10), then you could run a repeated measures ANOVA (Between = therapist category, x Within = pre- post-scores).

Stephen Politzer-Ahles , thank you for your response.

This research question is actually step 1 of the study. I do think it is relevant though in the discussion to what extent treatment effects can be attributed to differences between therapists, patients, or the specific therapeutic interventions.

The second step in this study is to examine whether certain activities of the patients (supervision, reading literature, etc.) may explain the potential differences between therapist. But first we need to know whether there are any differences between therapists (in our sample).

Jos Feys, thanks. If I understand correctly, your suggestion is to decrease the number of levels of the therapist factor, so that it can be used as a fixed factor in a repeated measures ANOVA? Sounds relevant.

However, I am not sure on what ground I could categorize therapists...

You don't need regression models to know if there are differences between therapists; just looking at the average outcome per therapist (in, e.g., a histogram or strip plot) will show you that.

You could categorize therapists on the basis of Gender, or Age, or Education, etc. Provided of course, that such categories are theretically relevant. (I am not at all familiar with the subject.)

Well, your second step seems to indicate what categories might be relevant: supervision, reading literature, etc.

Opposed to Stephen Politzer-Ahles I believe it is a pretty valid research question how much therapy outcome is associated to the therapist by yet unknown impact factors. In any case, this is best done with a multilevel model, e.g.:

library(lme4)

lmer(post ~ 1 + pre + no_patients + (1 + pre|Therapist))

More on multilevel modelling in my online book: https://www.researchgate.net/project/Book-New-statistics-for-the-design-researcher

Thank you all for your input!

Jos Feys, now I do understand what you mean. Have to think about it, because it combines my two research questions into one step... which is different from adding variables such as hours of supervision, etc. as additional variables into the model, next to the variable therapist. Hmm...

Stephen Politzer-Ahles I am not sure whether I agree. Of course, I can just calculate the means per therapist, and of course there will be some differences. However, I guess I need statistical tests to determine whether these differences are statistically significant, and even more relevant, what the effect size is. I.e., how many of the variance in outcome can be attributed by differences between therapists.

Martin Schmettow Thank you very much. I only know some basics of linear mixed effects models, but I do think that this is the most suitable way to analyze these data.

I agree with Stephen Politzer-Ahles

Frank Euteneuer thank you for your suggestion! I am not an expert on this, so I want to check whether I understand you correctly.

Is it true that what you propose is to perform a multilevel analyses with all single measures on level 1, in order to take into account that not all therapists have an equal number of cases? Combined with the suggestion to categorize therapists (on level 2) to increase power? And thus to create a fixed factor of "therapists" instead of a random factor?

In other words, do I understand correctly that what you propose is a combination of Jos Feys 's suggestion (make categories of the therapists) and Martin Schmettow 's suggestion to use a multilevel model?

Thank you for thinking along!

hierarchical model is synonym for multilevel model.

Martin Schmettow thank you again. I do know indeed that a hierarchical model is a synonym for a multilevel model (also called mixed models, linear mixed effects... etc.)

However, I do not quite understand what the benefit is of categorizing therapist in the multilevel model, when it seems - as you suggested if I understand correctly - that you can also add the 'original' therapist (random) variable into the model. The latter seems 'more correct' to me when taking the differences between fixed and random variables into account. Also, it would make it possible to first run a more simple model in order to estimate a general "therapist" effect, followed by a more complex model including other variables such as amount of supervision, etcetera.