Dear Colleagues
I hope all is well.
I have done machine learning model for prediction of survival among kidney patients.
My boss is keen to do subgroup analysis based on ethnicity to see if our prediction model metrics hold on minor ethnicities or no
My question:
To my understanding, I cannot do subgroup from the whole dataset as this will lead to data leak . Is that right?
In this case, the only way to do subgroup analysis is by taking a subgroup from the test dataset , not the training dataset.
Is that right?
You are correct that performing subgroup analysis on the same dataset that was used to train the model leads to the problem of data leakage (as well as some other problems). If I understand the goals correctly (to see if your model correctly classifies survival status among each of the race categories), then you could likely summarize prediction performance (e.g., % correctly classified) on each of the race categories found in your test dataset. Assuming no other issues, this will provide you with insights into how well the model is performing for each subgroup without the risk of data leakage. This could potentially lead to imprecise estimates for smaller subgroups, however.
- Badges
- Science topic