What are the methods to measure the bias and variance of a machine learning algorithm (regression) in order to improve it. I have designed a neural network for regression problem and I want to measure its bias and variance and measure the improvement after building an AdaBoosting ensemble.
Dear Nisar,
Searching for a good compromise bias / variance in machine learning is a laborious quest. There is currently no real consensus in terms of methods and tools to succeed in, but I suggest that you follow these references of interest to help you answer your question:
- Sharma et al., " Bias-Variance Tradeoffs in Program Analysis ", 2014 - https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bvt.pdf
- P. Domingos, " A Few Useful Things to Know about Machine Learning ", 2012 - https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf
- Arlot et al., " A survey of cross-validation procedures for model selection ", 2010 - https://hal.archives-ouvertes.fr/hal-00407906/document
- Ghosn et al., " Bias Learning, Knowledge Sharing ", 2003 - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.137.9813&rep=rep1&type=pdf
- Rahaman et al., " On the Spectral Bias of Deep Neural Networks ", 2018 - https://arxiv.org/pdf/1806.08734.pdf
- Bengio et al., " No Unbiased Estimator of the Variance of K-Fold Cross-Validation ", 2004 - https://www.hds.utc.fr/~grandval/nips03.pdf
- S. Raschka, " Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning ", 2018 - https://sebastianraschka.com/pdf/manuscripts/model-eval.pdf
- Shalev-Shwartz et al., " Understanding Machine Learning: From Theory to Algorithms ", 2014 - https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf
Regards
Basically, bias is systematic error of your model, which causes your model to do false predictions nearly all the time, due to incorrect modeling, unrelated data usage etc. You can use different regression models and calculate differences between predicted values and target values in order to assess bias.
Variance is the dispersion of predicted values over target values with different train-sets. Again, you can train your regression models with those train-sets but this time, you have to assess how do predictions vary through each train-set.
As you can understand, bias is about the model or data itself and variance is about model sensitivity to training observations.
Hazar Altınbaş Thanks for your answer. I have calculated the variance of the model using "leave-one-out" approach. I used all the data for training except one sample and predicted it, same is repeated on all samples and error is calculate between actual and predicted values for each sample. The standard deviation of this error vector is calculated and used as a measure of variance of the model.
For bias, I am still unsure how to measure it. I know it theoretically but can't formulate a way to calculate it. It will be appreciable if you can share some method or a reference.
Is it the good approach if I calculate the variance and subtract it from MSE and take a square root as in the attachment.
mse = (bias)^2 - variance
Bias/Variance estimation:
- [Equation 5] G.U. Webb, " Estimating bias and variance from data ", 2003 - http://i.giwebb.com/wp-content/papercite-data/pdf/webbconilione04.pdf
Selection Bias Correction:
- Cortes et al., " Sample Selection Bias Correction Theory ", 2008 - https://arxiv.org/pdf/0805.2775.pdf
- V.-T. Tran, " Selection Bias Correction in Supervised Learning with Importance Weight ", PhD Dissertation, 2017 - https://tel.archives-ouvertes.fr/tel-01661470/document
Dear Nisar Ahmed ,
I've found this draft paper (http://i.giwebb.com/wp-content/papercite-data/pdf/webbconilione04.pdf) and I think it is very easy to implement the approach proposed by Webb, through pages 7-9. In equation you provided above, I think irreducible error is missing.
Thanks for answers, I think I have found the solution after reading the links mentioned in your posts.
Ah I missed Mr. Breton's link and shared the same paper. Sorry for replication.
- Badges
- Science topic