Hi,
I generated several machine learning models with different algorithms and hyper-parameters and different preprocesses. My models always end up being systematically biased, as shown in the attached image done with xgboost (red is the 1:1 line ; blue is the linear trend of the bias). I could easily patch the problem with a linear regression, but I feel a more elegant solution could be applied upstream. Any idea?
Thanks!
https://towardsdatascience.com/balancing-bias-and-variance-to-control-errors-in-machine-learning-16ced95724db
https://towardsdatascience.com/preventing-machine-learning-bias-d01adfe9f1fa
https://arxiv.org/pdf/1908.09635
Article Learning from imbalanced data: Open challenges and future directions
I wonder if you have an omitted variable or variables, though you do not want to overfit. (Bias-variance tradeoff is a big concern, as noted above.) The nonzero intercept shows a constant bias, but I think intercepts are often just to cover missing variables. Just a guess though, as I worked with simpler models.
I don't see heteroscedasticity, though it is not always apparent on graphs. Also, with more complex situations, I think there is a tendency to lose naturally occurring heteroscedasticity (larger predicted values should have larger sigma of estimated residuals, otherwise) perhaps because of difficulties in having predicted y which matches qualities of real y. See https://www.researchgate.net/project/OLS-Regression-Should-Not-Be-a-Default-for-WLS-Regression. So if your model must be complex, without being overly complex, then perhaps you might not expect heteroscedasticity. Regardless, you can check it with this tool:
https://www.researchgate.net/publication/333659087_Tool_for_estimating_coefficient_of_heteroscedasticityxlsx
- Badges
- Science topic
- Similar topics
- Computer Science
- Images