There are two cases. Case 1 Use the benchmark data sets provided by other researchers and available online like UCI machine learning datsets..Advantage is that you may find many papers using these datasets and easy to compare the results...

Case 2: build your own data set and compare different algorithms through implementation or provided codes...In this case you have to justify that it is a good or novel benchmark data set...

Choice is yours :)

You can use different data sets or create your own dataset. Creating your own dataset will take time as you need it to be tested. However, it will increase your learning and you might reach to a new unknown point which was needed to be discovered.

First implement the current datasets to test. Then go on experimental research for development of new algorithm for your own machine.

We use the data sets from the UCI machine learning repository. Also, you could search for previous publications on the same machine learning topic (i.e. association rules, classifiers, etc.) and use some of those. It usually helps to get you published if you compare to more data sets and if you can show the results are better than previously published ones.

For data,as others have said, the UCI collection is a useful starting point. But there are many other more specialised collections, for example when learning to optimise something. Think carefully about what your audience care about. Many academic papers merely compete to outperform previous algorithms on standard datasets such as members of the UCI collection. For practical use, it can make more sense to seek out large, real-world industrial/commercial datasets and discover what the state of the art is for dealing with them, and se if you can improve on that.

For algorithms, it can be helpful to utilise some pre-existing ingredients at the start, if only to give yourself confidence that tour ideas have some value. But writing your own code can give you a much greater understanding, and control, than using other people's code. Yes, it may take a lot of time to write your own, but it also take a lot of time to properly understand somebody else's code. I have been surprised from time to time by bugs I have found in other people's code; I also create bugs by mistake on occasion, but it's much easier to test my own code if it's designed for the purpose, than it is to test somebody else's code that has a less-than-familiar design philosophy.

I rarely find GUIs helpful; all too often, working with somebody else's ideas of what counts as useful output is too much like keyhole surgery. If you write your own code, you can design the output to be highly flexible, of a kind that can be processed by general-purpose tools such as R or gnuplot or whatever.

To compare MVA's I have used the package TMVA http://tmva.sourceforge.net/ (I was one of the developers earlier at CERN). It is useful as it will test and compare your choice of MVAs with your given data. There are many MVA's to chose from as well as many data preprocessing methods.

However the tool was created in a particle physics community and it depends heavily on ROOT (a sw tool used at CERN, packed with stuff...). But, it has a lot of documentation and if one can get it to run on your own data set, it can be very useful. I have tried using it with success in my current job in bioinformatics.

The TMVA package is written in C++.

I have developed a new ensemble classifier for streams in my master's dissertation and I've used MOA to develop and test it. To benchmark it, I've used random generated streams, such as AGRAWAL, SEA, STAGGER, etc. When benchmarking a data stream there are at least 3 other things other than accuracy that you must assess, namely: processing time, memory usage and drift recover. It is a bit difficult to assess those if you use static data (databases that were originally created for traditional machine learning). Why is that the case? Well, if there is not enough data to "simulate" a stream, it will be difficult to assess memory and processing time. In the context of data stream mining, not enough data often mean less than 10 thousand instances. Also, if you use a static database you may not know for sure where and how concept drifts are. Therefore, it is difficult to assess concept drift recover (which is one of the most important things in stream mining). Even though, there are some "real" data that comes from a stream data source on UCI and you can use it, i.e. sensor data, computer network traffic and stock trading information, not all databases you find on UCI will have data stream characteristics (in reality, just a few will do).