I have trained a machine learning model to predict the outcome of a therapy based on some pretreatment information. Outcome is based on a questionnaire that is administered both before (pre) the treatment started and after (post) it ended.
As seen in other similar paper and challenges, I used as outcome of the machine learning model the post-pre change score. It should be also of primary importance to clinicians. Then I reconstruct the post score = observed pre score + predicted post-pre change score.
If I calculate the (nested cross-validated, outer loop) r-squared with the classical formula (not the square of the correlation coefficient) I obtain r-squared post score < r-squared post-pre change score. Nothing strange as to calculate them, the numerator is the same but the denominator is different, with the variance of the post score smaller than the variance of the post-pre change score.
However, I’m quite confused about which of the two, either the r-squared post score or r-squared post-pre change score, is the most useful metric to judge how good my model would be in practice.
Thank you,
Massi
I think any flavour of r² is not quite a useful metric in practice. I think a clinician would like to know a credible range of the post-score, or possibly a credible range of the score-change. Depending on the application and other things the clinician has to consider, it may be good to have ranges for several degrees of credability, like 50%, 90%, and 99% or so. I think that would be most useful.
The difference between post-pre score and post score is that the former takes into account baseline variation, while the latter doesn't.
This means that the post score is a combination of 2 sources of variation - the baseline score and the change induced by the intervention.
A regression model on the effect of intervention would be expected to give zero information on baseline status (unless some bias is present). This means that post score increases the total variance but adds zero explained variance, leading to a lower R2 as you observed.
This will then apply to Jochen's answer. I am not sure why you are interested in R2 either, but statistical tests are based on comparing variations and so post score will always be dirtier than pre-post scores due to the mixing of (hopefully) independent sources of variation in the former.
Thank you Jochen! Your are absolutely right for a clinical point of view.
And thank you James for the explanation of difference between the two R2!
- Badges
- Science topic