Suppose we have a Neural Network with a binary output (0 or 1). What I am trying to do is to remove neurons or layers from the NN while maintaining a correct classification for all the instances that were classified as 1 in the original NN, same thing for the output 0. Said differently, is there any way to spot neurons that are paramount to the correct classification of the instances of a particular class ? The aim is to remove all the unnecessary neurons regarding that output. Currently, I am trying to use the back propagation phase to try to attribute a fitness to each neuron regarding its contribution to a certain class.
In the case of Binary Neural Networks (binary weights and activations), a research track could be compiling the NN to a Boolean Formula and reasoning on it to spot the neurons that does not contribute to the chosen output, but it is not always obvious to carry out this compilation.
There are a few ways of achieving something like this:
1. You can use dropout layers Article Dropout: A Simple Way to Prevent Neural Networks from Overfitting
and allow the network to figure this out by itself.2. You can use dropconnect layers Conference Paper Regularization of Neural Networks using DropConnect
and allow the network to figure this out by itself3. You can use feature importance algorithms to detect which features are actually be important. This can help you fine tune the size and architecture of the first layer.
What you are looking for seems close to weight pruning. The idea is to get rid of parameters that are close to zero as they should have low impact on the results.
It is possible to do that during training so that the model continue to improve its performance while reduing its number of parameters.
https://www.tensorflow.org/model_optimization/guide/pruning
https://medium.com/tensorflow/tensorflow-model-optimization-toolkit-pruning-api-42cac9157a6a#:~:text=What%20is%20weight%20pruning%3F,layers%20of%20a%20neural%20network.
Weight decay is what you are looking for!
Weight decay is an optimizer hyper-parameter that controls the amount of learning factor of each neuron. So during backpropagation larger and lower weights are penalized pushing lower weights to 0. This allows your model to produce less overfitting because larger weights are penalized, also in your network are activated only a neurons dedicated to specific features.
For such neuron pruning, I suggest you to follow the link https://au.mathworks.com/matlabcentral/answers/361777-prune-hidden-neurons-in-neural-network
Also follow https://au.mathworks.com/matlabcentral/answers/110464-how-to-remove-outlier-from-training-data-before-training-artificial-neural-networks
1st check for symmetry
2nd try to apply dropout
But before applying dropout you have to make sure that yr back propagation produces correct result by cross varying with the monotonic decreasing fashion of cost decay.
I have found a pretty interesting study carried out by the NVIDIA team for network pruning. Given a trained network N and a dataset D, the method is as follows:
1. Rank the neurons according to their importance: Using a first order Taylor expansion of the network cost function, the rank of a neuron is dependent on its activation and gradient during the forward pass (multiplication of the two).
2. Remove the least important neurons
3. Fine-tune the network
4. go back to 1 or break
The choice of the dataset D could determine the kind of neurons we want to keep/throw. For instance, if I work with a dataset containing just the instances of a certain class, I will remove the neurons that do not contribute to the correct classification of this class during inference.
Link to the paper: https://arxiv.org/pdf/1611.06440.pdf
- Badges
- Science topic