Your sample size directly depends on the quality and size of your samples.  More is not necessarily better.  For instance, one comprehensive sample that contains many interesting features may meet your needs--especially if the learner decomposes the sample.  Many small samples that don't contain much variation may be a poor sample set.  Machine learning depends on variations, relations, and your topical interests.  Unless you are imposing some kind of frequency criteria (which can be performed by feeding the same sample multiple times), collect and/or generate a set that includes enough variation in the area of interest.  Also, test your learner 1) against the same set to verify correctness, 2) against bad or unlearned examples to force failure and check performance.

Thank you Mostafa,

You are right and more data is better because in this way we can create better models and evaluate our models using learning curves much more better.

But I should talk about this issue (the number of samples) in the research proposal and provide an exact information. 

Your sample size directly depends on the quality and size of your samples.  More is not necessarily better.  For instance, one comprehensive sample that contains many interesting features may meet your needs--especially if the learner decomposes the sample.  Many small samples that don't contain much variation may be a poor sample set.  Machine learning depends on variations, relations, and your topical interests.  Unless you are imposing some kind of frequency criteria (which can be performed by feeding the same sample multiple times), collect and/or generate a set that includes enough variation in the area of interest.  Also, test your learner 1) against the same set to verify correctness, 2) against bad or unlearned examples to force failure and check performance.

For datasets, an area of research that I'm currently pursuing, there's a KDD repository for all kinds of research.  You may also be interested in using an open source tool Weka that includes machine learning algorithms, sampling techniques, and dataset preprocessing filters.  I've also include a link to Sharon Lohr who publishes many articles and books on sampling techniques where you can read further into evaluating which technique is better depending on the dataset and its context.  Also I included link to a dataset generator that can create sample datasets and export it in many formats.  Hope this helps.

http://archive.ics.uci.edu/ml/

http://books.google.com/books/about/Sampling_Design_and_Analysis.html?id=aSXKXbyNlMQC&pgis=1

http://www.cs.waikato.ac.nz/ml/index.html

https://webapp4.asu.edu/directory/person/11072?pa=true

http://www.generatedata.com/

it depends on your goals. You need a dataset which has got a set of instances that suitably represent each value of your target attribute.  You need that each class is more or less balanced. More instances allow you to build different subsets of instances and check if your mining models are robust or, otherwise they are over-fitted

I'll add a simple suggestion to the other helpful answers: do it iteratively, i.e., every N samples added test performance accuracy on a simple machine learning task until you've reached a result that suits your goal.

The size of your data set essentially depends on the data that is actually available to you. If you have a lot of data, you may use n-fold cross-validation such as the popular 10-fold cross-validation. Or if your data is scarce, you may want to perform leave-one-out cross-validation as to maximize the usage of the limited data. It is also good practice to set aside, say 15-20% of the data, as an external validation.

I agree with Keven that a higher degree of variance imposed from the data is important for the machine learning algorithm. Perhaps, you can use Principal Component Analysis (PCA) to see the relative distribution of samples in your data set. You may want to read up on Statistical Molecular Design from Anna Linnuson for ideas.

Article Statistical Molecular Design of Balanced Compound Libraries ...

There is no second opinion on saying that larger data set would give you better results. However, it all depends on your application, if it has the capability of learning than your case base or data base would enhance it self by adding results from your solutions. Your application would become intelligent as you use it.  However, there is no limit to a specific number for a data set to start with.

If you are using a face to face questionnaire to collect your data (as we often do in Agricultural studies), then you can estimate the sample size mathematically. The size of required sample are given by Cochran formula [G.W. Snedecor, W.G. Cochran, (1989). Statistical Methods, 8th Ed. United States:Iowa State Press, 1989]:.

n=Nt2S2/(Nd2+ t2 S2)

where n is the required sample size; N is the number of holdings in target population; t is the reliability coefficient (1.96 which represents the 95% reliability); S2 is the variance of studied qualification in population;and d is the precision.

N.B. The above formula give you an estimate on minimum number of samples for conducting your research. The ML task is a different story! All ML tasks are data-driven so the more examples you use to train a network the better you get to the underlying mapping between input-output.

As a statician, I can not agree with "the greater, the best" rule.

It all depends of the parameter(s) you want to estimate: mean, standard deviation, proportion,... or some combination of them.

There are probabilistic rules to select your sample size depending of the maximum sample error you wish to impose to your estimations.

Regards

Chanin raises the important point of cross-validation. The reason this is important is because it allows you to test how well your model generalizes to new data. Cross-validation is particularly useful when you don't have much data in the dataset, but you should always check how well the model generalises even if you have lots of data. In the latter case you should do something like an 80:20 split of train:test.

Can you explain more about what data you're collecting/generating and what machine learning you're wanting to do Sajjad? Some techniques inherently require more training data than others (e.g. Conditional Random Fields require more training data than Hidden Markov Models, which allows them to represent more complex processes).

Regards,

Rolf

You may want to consider how psephologists work. If you have a complex feature space you could set up a sampling frame and then randomly sample within cells in the frame. This improves your chances of meeting the represenatition criterion with smaller samples. I used this technique in http://dl.acm.org/citation.cfm?id=2537046&CFID=451633107&CFTOKEN=68920688 (Gold Open Access - free download).

You can assess the  size of your data set by evaluating F-score behavior for each instances number of your dataset. Herein, you analyze how the learning procedure can be influenced by the instances number, you can compute a learning curve, by dividing your data into different learning sets. For each set, we apply the learning algorithm. To avoid the over fitting, all the reported experiments should done with fold crossvalidation on our entire data-set.

Thank you all for your beneficial comments, they were very informative and practical. 

I think maybe there is no exact answer  to this question and maybe it's better to do it iteratively.  

Dear Rolf,

Actually the data is about heart coronary diseases and risk factors that lead to this kinds of diseases. At the moment I do't know more about the exact features. I'm working with a heart fellowship and he is suppose to prepare a questionnaire form. He wants me to specify a minimum number of instances for our data set. Maybe because he is not familiar with machine learning concepts he prefer to have a specific methods to define a minimum number of instances.

Best,

Hi Sajjad,

Thanks for clarifying. Yes I agree with your conclusion, and it's going to be pretty hard to define the minimum number of instances because it really depends on what the features are and what machine learning technique you use. I'm guessing he wants to know the minimum number of people he has to send the survey out to or something?

Hello Rolf

Thank you Rolf for your comment,

You are right ,we need to write a proposal and send it to a related institution. And one part of this proposal requires the number of samples. 

To state it simple, the following things must be kept in mind when generating data sets.

Keep the distribution of attributes in mind. There would be no use of having an attribute in a data set if only one or two of it provides different value while the remaining 100 or so gives the same value (unless the attribute is a conctrete determining factor for the decision).

Try to have sufficient negative and positive data.

Have sufficient data so that it would provide tangible results when split according to the 60:20:20 rule for Training:CrossValidation:Testing.

Based on behavior, ancillary information like age, gender, color, gesture etc or based on epoches you can create different clusters of data...