It is true that the sample size depends on the nature of the problem and the architecture implemented. But, on average, what is the typical sample size utilized for training a deep learning framework?
For instance, in a convolutional neural network (CNN) used for a frame-by-frame video processing, is there a rough estimate for the minimum no. of samples required to train the model?
It really is pretty hard (if not impossible) to give a 'one size fits all' answer. The amount of training data you require is dependent on many different aspects of your experiment:
- how different are the classes that you're trying to separate? e.g. if you were just trying to classify black versus white images then you'd need very few training examples! But if you're trying to solve ImageNet then you need training data on the order of 1000 examples per class.
- how aggressively can you augment the training data? (Sander Dieleman's blog post on 'Classifying plankton with deep neural networks' gives a great introduction to data augmentation; and he talks a little about the question of 'how much data do deep neural nets require')
- can you use pre-trained weights to initialise the lower layers of your net? (e.g. using weights trained from ImageNet)
- do you plan to use batch normalisation? It can help reduce the amount of data required.
- There are loads of other 'tricks' discussed in the literature on ways to squeeze maximum advantage from a small training dataset.
http://benanne.github.io/2015/03/17/plankton.html
It really is pretty hard (if not impossible) to give a 'one size fits all' answer. The amount of training data you require is dependent on many different aspects of your experiment:
- how different are the classes that you're trying to separate? e.g. if you were just trying to classify black versus white images then you'd need very few training examples! But if you're trying to solve ImageNet then you need training data on the order of 1000 examples per class.
- how aggressively can you augment the training data? (Sander Dieleman's blog post on 'Classifying plankton with deep neural networks' gives a great introduction to data augmentation; and he talks a little about the question of 'how much data do deep neural nets require')
- can you use pre-trained weights to initialise the lower layers of your net? (e.g. using weights trained from ImageNet)
- do you plan to use batch normalisation? It can help reduce the amount of data required.
- There are loads of other 'tricks' discussed in the literature on ways to squeeze maximum advantage from a small training dataset.
http://benanne.github.io/2015/03/17/plankton.html
case by case it could be different, there are some cases you may not be able to even collect 30 sample for training the system like if the research is on tsunami. Hence, there is no minimum or maximum for training sample size. Generally the more sample for training can ensure better performance of system but make sure dont get over trained your network.
According to Yaser S. Abu-Mostafa(Professor of Electrical Engineering and Computer Science) to get a proper result you must have data for at-least 10 times the degree of freedom.
example for a neural network which has 3 weights you should have 30 data points.
These problems are entirely dependent on the domain and the quality of your data sample. I do remember, in the past, seeing a comment that a moderately complex problem requires about 1,000 training cases for a decision tree (from Quinlan, around the ID3 period).
This is very much a research topic. The factors that determines the sample size are:
The statistical characteristics of the data depends on the distribution from which it selected. Assuming different domain have different distribution of data, it could be said that the number of training samples required is domain/application dependent.
However, I must say... it is not like you should have 10 times the degree of freedom... It is not true! When there is more data than required, the classifier either tend to overfit or just dis-regard the samples. Probably, he might have given this rule of thumb to save some effort for his students...!
There is some good articles that described ways of using deep learning models when the data set is small:
https://beamandrew.github.io/deeplearning/2017/06/04/deep_learning_works.html
https://medium.com/merantix/applying-deep-learning-to-real-world-problems-ba2d86ac5837
https://beamandrew.github.io/deeplearning/2017/06/04/deep_learning_works.html
is performed on MNIST and devolves in to a problem that can be solved with a linear classifier as identified in the blog itself.
In practical cases, more data is needed for training a deep model since you want to see the model’s ability to generalize. No one can tell you how much data you need because the answer is " It is unknowable" the reason why of the complexity of the problem and the model implemented.
The size of the data is not very important. What is important is the variations in the samples you have. That is why data augmentation increases the accuracy and the generalization of a CNN model. However, you need to have additional training data if your model is over-fitting. If your model does not over-fit with n samples, you only can get benefits of additional good quality data in improving the generalization of the model. In other words, you need additional data if you need more accurate model. If you are just fine with an accuracy of 80%, thus just get n samples of training data that achieves that accuracy.
Apparently you're not going to get a rule of thumb as there are too many variables involved. However, I have been training several CNNs over the last few days for the purpose of steering from camera input. 'Behavioural cloning'. The models are approximately 5 million parameters in size and have a single regressed output. I want to use the minimum amount of samples I can for the purpose as collecting appropriate data is tedious. I trained models with about 40, 60, and 80 thousand samples (16 epochs). Each exhibiting marked improvement on the last. At 80 thousand samples the models look like they are just starting to do their job as intended. I'm about to start training on 140 thousand samples and expect to see significant improvement. I suspect 1 million samples would do very nicely.
I should however say that if I collected a few thousand samples from every country in the world and it added up to a million samples, then trained my model with that data what would it be good for? It would not be particularly good in any country. If I train my model on a million samples collected from London, it would probably be quite good in London and numerous other British cities. The sample data and application relationship matter, despite generalisation capabilities.
Obviously we want good generalisation, but many real world problems are just too complex for a single model.
Focus on your objective and this dictates everything including your sample size. Lets assume that your objective is to produce a variety of outcomes, the sample size should be able to include all of the possible outcomes expected out of your neural network model. This is very crucial. Remember neural network is only as intelligent as your samples and training algorithms. Accuracy and precision depends on your sample size, its heterogeneity. As suggested, there is no magic number but there is a rule where "sample should adequately represent the population". Selection of sample sizes could be guided by the distribution of population and their behaviours. If this unknown, go for random selection until you reach a minimum error metric. This is an iterative process.
it is not only the size of data but also what is percentage of each class.from my experience as much as you increase the data size it gives better results.
if you don't have enough size you may increase it using technques such as data augmentation. If you sample classes is unbalanced you can start training with balanced number the complete training with the rest
In order to find out how good your dataset size is, one measure is over-fitting. try to classify your data using training set and then repeat the classification using cross validation.If you increase the data size it gives better results in CNN classification process's.
The optimum input value to get specified output it to put output as input, one layer identity matrix and no non-linearity, but I assume that is not your question.
If you already have a limited and specified set of input data that you would like to use in order to predict the output, then one way to go is to just use it as input directly. One of the strengths of neural networks, especially deep ones is to learn meaningful features (that are good predictors of output values) fully automatically (by some gradient descent optimization).
However, that is not to say that sometimes manual feature creation might be quite beneficial to your task.
Just to give you a simple example, if you had input data in 2d euclidead space and wanted to predict the distance from origin (x^2+y^2=d^2) it would be very beneficial to put the squares of x and y as input to the neural network, but you can get similar results with a more sophisticated deep net and simply x,y as input.
It really is a hard question that I am interested so much. But where is the answer?
A discussion on
https://towardsdatascience.com/how-do-you-know-you-have-enough-training-data-ad9b1fd679ee
A reference: https://arxiv.org/pdf/1805.07883.pdf
VC Dimensionality decides the sample size for all the classifiers. And CNN is no exception.
Shankar Lal , hi could you please elaborate on it. as For an ordinary neural network, the VC dimension is roughly equal to the number of weights. So I should have images(lets take image classification as the underlined task) equal to number of weights :/ ?
The Vapnik–Chervonenkis (VC)-dimension formula for neural networks ranges from O(E) to O(E2), with O(E2V2) in the worst case, where E is the number of edges and V is the number of nodes. The number of training samples needed to have a strong guarantee of generalization is linear with the VC-dimension.
Refer https://igi-web.tugraz.at/PDF/139.pdf
for extensive discussions on the topic.
- Badges
- Science topic