Hi, I have a set of activity data for compounds that can be active or inactive against a specific protein. I am going to make a model using binary classification models like Random Forest. I am wondering if this is possible to add uncertainty to the binary classification model?
Thanks
I refer to this article https://arxiv.org/ftp/arxiv/papers/1711/1711.09677.pdf
https://stats.stackexchange.com/questions/299340/uncertainty-in-binary-classification-of-new-data-via-random-forest
http://shftan.github.io/surfin/demo.html
What do you mean by "uncertainty"?
uncertainty in the classification, in the ROC curve, in the misclassified samples?
Is it binary or multicategorical?
If so, it is a common mistake to use uncertainty statistics from confusion matrices: certain measures are rarely normally distributed.
The statistic you compute HAS to take into account model variability, so if you compute it using for instance a confusion matrix using caret for instance, it does make little sense, since they take the model as fixed.
A good practice in machine learning is to perform resampling, i.e. split into training/test and compute the statistic of interest in the test set,
replicate 100-1000-10k times, then plot the distribution and compute values of interest, i.e. mean, median, mode, 95% interval and so on.
With Random Forest it is particularly simple because you can have a model trained with no fuss then replicated it N times.
Stefano Nembrini thanks for your reply.
I meant uncertainty in the predicted class. Lets assume that my SVM model classified compound A as an active compound (i.e., active class). I want to know the level of uncertainty for this prediction.
Off course I can assess the model with other parameters like accuracy, recall, Cohen's Kappa and etc but I want to add uncertainty to the predictive model to see how much (un)certain is the predicted class for each compound.
Those are not parameters in a statistical sense. I am still not sure what you mean because accuracy and "uncertainty in the predicted class" are the same thing for me
The caret package in R offers different measures that can be replicated with a bootstrap or replicated cross validation
This is usually how uncertainty is assessed in the machine learning world
I concur with Stefano Nembrini . If you train the model on 10000 samples and predict 1/0. The OOB-error or cross validation results in a model accuracy of 80%, then 80% could be regarded as the "certainty" and 20% as "uncertainty". Or at least, the model (training) classified 80% correctly and 20% not, based on the chosen methodology. The model prediction was thus "wrong" 20% of the time. By accepted ignorance or literature you could suggest that +/- 80% is relatively normal. The question remains if the model performance is similar applied on completely different dataset (which is actually the most interesting part). While this is relatively straight forward for spam e-mail (since everyone get this), for more "complex" data this is less easier obtained and testing the model might be difficult.
The caret package also returns the confidence intervals, but this is not the uncertainty. If you have >10.000 samples the confidence intervals are simply +/- 1% anyhow, but does not tell you how the model would perform on other data.
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification