Hi, now I am dealing with a picture with size of 72*72 and I want to use the Convolutional Neural Networks to classify by the label. I know there are mant differnt ways to design the size of convolutional kernel and sampling map. However, the accuracy is just 82%now. Are there any rule of thumb about how to choose the size,or layer or kernel? I can just apply the model on my personal computer,so it could not be too deep. I hope you can give me some advice! Thank you very much!
There are many tricks you can use in ConvNets. A good advise is to increase the network size until it overfits (on the validation set) and then add regularizers and data augmentation and if it doesn't overfit any more, again try to increase the model size. Generally, the model size depends on the amount of data and its complexity, but if you use max-pooling, you want to increase the number of neurons towards the top (e.g. double for each layer). Usually it will be 2-5 layers before the dense layer and for the kernel sizes its trial and error (somewhere between 3 and 5). To get a feeling for what works best, you can do a grid search over the parameters.
Some regularizers:
- Batch normalization can help if you have more layers (>2). It prevents diminishing gradients (all layers receive the gradient to a similar extend). http://arxiv.org/pdf/1502.03167v3.pdf
- Max-Norm regularization & Dropout: http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf
- L1 / L2 weight regularization
- Sparsity regularization (e.g. http://web.eecs.umich.edu/~honglak/nips07-sparseDBN.pdf)
- Gradient clipping (more thorough search in the cost landscape)
- Data augmentation increases your dataset and thus prevents overfitting (and you can inject prior knowledge) and max-out units were successful in recent image-classification contests:
http://benanne.github.io/2014/04/05/galaxy-zoo.html
http://benanne.github.io/2015/03/17/plankton.html
how many labels are in your dataset? 82% might be quite good.
CNNs tend to require lots of empirical experimentation to find the optimal parameters. This is why it is so important to have a validation set so that your results aren't biased.
There are many tricks you can use in ConvNets. A good advise is to increase the network size until it overfits (on the validation set) and then add regularizers and data augmentation and if it doesn't overfit any more, again try to increase the model size. Generally, the model size depends on the amount of data and its complexity, but if you use max-pooling, you want to increase the number of neurons towards the top (e.g. double for each layer). Usually it will be 2-5 layers before the dense layer and for the kernel sizes its trial and error (somewhere between 3 and 5). To get a feeling for what works best, you can do a grid search over the parameters.
Some regularizers:
- Batch normalization can help if you have more layers (>2). It prevents diminishing gradients (all layers receive the gradient to a similar extend). http://arxiv.org/pdf/1502.03167v3.pdf
- Max-Norm regularization & Dropout: http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf
- L1 / L2 weight regularization
- Sparsity regularization (e.g. http://web.eecs.umich.edu/~honglak/nips07-sparseDBN.pdf)
- Gradient clipping (more thorough search in the cost landscape)
- Data augmentation increases your dataset and thus prevents overfitting (and you can inject prior knowledge) and max-out units were successful in recent image-classification contests:
http://benanne.github.io/2014/04/05/galaxy-zoo.html
http://benanne.github.io/2015/03/17/plankton.html
- Badges
- Science topic