If I have 1000 class 'A' and 300 class 'B' samples, and train a network over the data and get a MSE of 30 (the network is trained for specific number of epochs), is there any link between the threshold and the error? Can the threshold for binarizing the output for the test set be calculated from the error + class ratio?
Take a look at the ROC curve (Receiver Operating Curve). This method enables ones to set the optimal threshold on the basis of the desidered FP rate and TP rate.
hi Safi, i think that to answer this question is better to have some more details. binarizing the output of your network you mean? which kind of activation function you use for the output neuron?
thank you Trincavelli, Yeah I want to binarize the out put. and the activation function is I am using tanh
Why don't you just simply try different threholds and calculate a correlation between them and the MSE?
yeah, that is for usre an option. Howver if you would find an optimal threshold value which is very different from 0 I would investigate whether the training was succesful or something went wrong...
Cheers
Take a look at the ROC curve (Receiver Operating Curve). This method enables ones to set the optimal threshold on the basis of the desidered FP rate and TP rate.
Threshold has direct impact on the error since based on the threshold the samples are assigned to one of the classes. In binary classification (assume two classes C1 and C2) if the output node takes values in the [-1,1] interval, which in your case, the hyperbolic tangent values fall in the [-1,1] interval. Then threshold/cutoff of B (e.g. 0.4) means if the output value of a predicted sample is bigger than B than it is labeled as class C1 otherwise class C2. So you can tune the threshold to get a reasonable performance both in terms of specificity and sensitivity. Mostly what you tune is this but in a more conceptual level you can consider the threshold not only for the output nodes but also for all nodes/neurons in the network.
Threshold has a great impact on the topology of your neural network, based on the threshold neurons get on or off which means that if the value of activation function of a neuron exceeds the threshold, it outputs a quantity otherwise it will be off. Therefor the number of active neurons and their connections can be manipulated by changing the threshold(theta).
Regarding the class imbalance issue if the samples in the smaller class are more important or more costly to generate or just sparse using the threshold you can put them in favor. For example when you put the threshold to '0' the probability of a sample classified as class A and B is equal but as you from the beginning the samples of class B are less abundant you can fix the threshold smaller/lager for class B. As I said you can run the classification with different threshold and see which threshold generate the most accurate classification results. But aware of over tunning and use cross validation methods before fixing your final parameters.
The error in the classification depends on the separation between the probability distribution functions in the n- dimensional feature space of the samples. The classification error occurs when there is overlap distribution functions between the two classes.
ROC is the standard way of analyzing this. The binarization threshold (aka decision threshold) defines two types of performance measures: true positive rate (percentage of the samples when the actual decision should be 1, and the binarization threshold delivers indeed a 1 ), and false positive rate (percentage of the samples where actual decision 0, and binarization threshold delivers 1). If you iterate for different values of the binarization threshold you obtain different TPRs and FPRs, which you can display in a graph TPR vs FPR. This is the ROC curve.
See work by Fawcett - ROC Graphs: Notes and Practical Considerations for Researchers, or it paper in Pattern Recognition Letters, "An introduction to ROC analysis". Very nice works!!
of course there is different answer for a Network with different threshold values.So the error will change. You must find a threshold value with the least error
i assume it is multidimensional inputs. yes, choice of threshold has direct impact on the results / error. what are other consideration factors for the end results? i had used ROC to evaluate and optimise my model so it can work well for accuracy performance and low false positive alarm rate. it is tendious but it is systematic approach to adjust the threshold. Hope it helps.
training of net with imbalance data of two classes must be done with robust training against imbalance. You continue solving this problem with training net with imbalance data set
I just wanted to mention that you need to be careful during training since your dataset seems biased towards class A. Usually training algorithms assume both classes are equally represented, you may want to fix a prior probability of a data vector being in Class A or B..
Nafiseh, in my humble opinion, cost of log likelihood ratio (Cllr) is much more powerful and consistent summary of confusion matrices (obtained from all possible thresholds AND class imbalances), than that Index of Balanced Accuracy (IBA), which these Spanish scientists keep on proposing in recent years (since 2007, I believe). =]. In this light, IBA looks like a toy solution, while Cllr was also generalized to multiclass case to obtain multiclass cross-entropy (Cmxe). More info on this here: http://arantxa.ii.uam.es/~jms/seminarios_doctorado/abstracts2007-2008/20070226NBrummer.pdf
Evaldas, I believe that you have not understood the meaning and purpose of the IBA measure when you are saying that this is a toy solution. Probably you should read our papers once again in order to gain insight on this measure, which has been proposed with the aim of overcoming the bias towards the majority class in an imbalance problem.
I'm wondering about the intent behind your question, I don't read your question as directly asking about thresholds in the ROC sense of a fixed linear discriminant function and a performance trade-off, but in the sense of determining the impact of say, a non-symmetric threshold used to binarize the output, and the use and effect of that threshold on training and convergence. Correct?
If the latter, correct classifications would not produce any training feedback of the network, so in the limit, where the threshold is set to one limit or the other of your sigmoidal limiting function, no training will be produced for one class or the other. In effect you introduce a cost function, emphasizing one class over the other. In that situation, MSE at a fixed number of iterations, I think, would have more to do with the complexity of the classes in your feature space than anything else.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence