I'd want to calculate the error bar for each data point resulting from my neural network model prediction. I'm seeking the most efficient method for calculating the uncertainty associated with each prediction value. Normally, we use the root mean square error or Standard Error Mean to evaluate our model's error, but I need to find an error bar for each point predicted by the model related to its actual value.
Soroush Sarabi My response is slightly lengthy, so as to be self contained. Please read through it and message me directly if you have additional questions.
First of all, if you are looking for error bars on the predictions of neural networks, that is not possible. However, if you are looking for prediction intervals (you might refer to them colloquially as confidence intervals), then there are methods to generate those.
You haven't mentioned the nature of the problem, the layer architectures, etc, so I shall use the one criterion that you have mentioned: "most efficient method". With neural networks, most of the time is spent in the training, so we'll select methods that minimize the computational expense in training.
There are many algorithms to generate Uncertainty estimates for neural network predictions. These range from pure Bayesian approaches, non-Bayesian approaches and hybrid approaches. Examples include Bayesian neural networks using HMC, approximate VI; ensembling, etc. With your criterion in mind, the method that can give you satisfactory Uncertainty estimates with minimal additional computational expense would be Monte Carlo Dropout (also know as MC Dropout).
In MC Dropout, you train a deterministic neural network with Dropout layers for regularization. Now, while under normal circumstances, the dropout layers are only active during training, in MC Dropout you force them to be active during prediction as well. Using this trained neural network, with active Dropout layers, you make multiple predictions for every new sample (usually 20-50 such forward passes are sufficient). This gives you a set of predictions made by the (stochastic) neural network for your sample. You can compute statistics over this set of predictions to choose the Uncertainty measure that you feel comfortable with. For instance, lets say you made 30 predictions for a new sample point in a regression problem. You can calculate the mean and the standard deviation over these predictions to get the mean and standard error of your neural network's predictions.
You can google MC Dropout and will find excellent tutorials on it, along with the original articles by Gal and Ghahramani, etc. Yarin Gal's blog and his thesis have detailed discussions on the method.
Hello (Salaam) Soroush,
Since ANNs have not a unique answer for the optimized weightings and biases, if we try to find the predicted values by running the ANN training many times we will get different values by which we can calculate a mean, an STD and the corresponding confidence interval and an error bar per each point.
- Badges
- Science topic
- Similar topics
- Statistics
- Time Series