What is the definition of the robustness of a machine learning algorithm? Is it different from the definition of the performance?
Dear El Sayed Mahmoud,
The robustness is the property that characterizes how effective your algorithm is while being tested on the new independent (but similar) dataset. In the other words, the robust algorithm is the one, the testing error of which is close to the training error.
For details see the following presentation:
( http://www.colt2010.org/presentation/rob_colt.pdf )
There you can find the references to the original papers as well.
P.S.:
To my best knowledge, "robustness to noise" (or "noise robustness") is a slightly different term, that describe the stability of the algorithm performance after adding some noise to your data (sorry for a bit self-evident definition=))
As Alexander Lebedev nicely described above, the robust performance of the algorithm is the one which does not deteriorate too much when training and testing with slightly different data (either by adding noise or by taking other dataset), hence, algorithm is prone to overfitting. To my best knowledge, this robustness property is also known as algorithmic stability. See a recent discussion about 'sparsity' vs 'stability' and how feature selection should be taken with caution when trying to improve performance of machine learning algorithms (which already have built in regularization) - Huan Xu and Shie Mannor "Sparse Algorithms are not Stable: A No-free-lunch Theorem":
http://users.ece.utexas.edu/~cmcaram/pubs/XuCaramanisMannor.NFL.pdf
By "Robust Algorithms", in a simple sentence, we mean those algorithms which have the ability to deal with the errors occurred at the time of execution and also, they can deal with erroneous inputs and parameters. For example,if we talk about "Image Segmentation" algorithms, then a robust algorithm will be the one whose output will have Low MSE(Mean Squared Error) values and High PSNR(Peak Signal to Noise Ratio) values. These two parameters are related to Noise measurements in Segmented Images. Generally, errors in case of image analysis are the noises that come to occur which mislead the further analysis process. So, a robust image segmentation will be the one which has the capacity of dealing with noisy images and whose segmentation results contains as low noises as possible.
In my knowledge,
-Robustness of a machine learning algorithm has low training error and low testing error.
i.e. low bias and low variance.
-training error about to close to testing error.
- Badges
- Science topic