When we compute the p-value for the Mondrian Conformal Predictor, for a given label, we compare the non-conformity score of the new test data with only training instances with that label that we are testing.
This is, given a training set T={(x1,y1)....(xn,yn)}, two classes Y={0,1}, and the new test data xn+1. When testing class 0, for instance, I'm interested in the non-conformity score of the examples: xi belongs to {1,...,n+1} given that yi=0.
My question is, to select yi=0, is yi the original class in the original training set, or is it the class given by the classifier? In other words, should I look the classes outputted by the classifier to select the training examples classified as 0, or to the original classes?
Thanks a lot.
.
my understanding is the following :
- we split the instances in a training set and a held-out set
- we train the model on the training set
- we deploy the model on the held-out set and compute the non-conformity scores
(therefore, non-conformity scores are computed from a trained model on out-of-bag (held out) samples, using the true labels of the held-out samples)
when we have a new sample, we predict its label, compute the corresponding non-conformity score and compare this non-conformity value with the non-conformity values of the samples of the held-out set which have a true label (not predicted label) equal to the predicted value for this new sample
(therefore, the selection is made on the true label value of the held-out samples)
.
usually, with a new sample x, we rather directly compute its non-conformity scores for each potential class
assume the trained model allows this computation for all classes, for instance in the form : a(y, x) = 1-Prob(y|x) for all y values
assuming y = 0 or 1, for our new sample x, we compute :
a(0, x) = 1 - Prob(y = 0|x)
a(1, x) = 1 - Prob(y = 1|x)
then we compare a(0, x) to the non-conformity scores of the held-out samples with true label y = 0, and compare a(1, x) to the non-conformity scores of the held-out samples with true label y = 1
.
.
see
https://arxiv.org/pdf/1603.04506v1.pdf
Conformal Predictors for Compound Activity Prediction
Paolo Toccaceli, Ilia Nouretdinov and Alexander Gammerman
.
There should be original classes for examples 1,...,n, and the testing (hypothetical) class for n+1.
- Badges
- Science topic