For a set of training data how can we analyse which algorithm fits best to get the optimized solution?
You should partition your dataset in a training/validation set and test set. You will use the training/validation set to trim the model parameters and find the best performing algorithm and then give as input to the trained model the test set in order to estimate the generalization capabilities of the model, such as misclassification rate for classifiers, or mean error for regression and so on.
I think a good partition of data can be 30/30/40
If you don't have a lot of data you sholud perform a procedure of k-fold cross-validation, see http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29
In this case the partitioning can be 60/40
Michele is it not too complex or time taking on triming each traning data to fit the model for all the available algorthims. Like in thelink provided they are considering only two models for optical character recognition. What does it limit to choose only SVM and KNN.
What I would like to know is there are any classification that for these type of problems this would be the better algorthim .
Comparing several algorithms and several parameter setting for each algorithm may indeed be computationally expensive. So the answer depends on the meaning of your original question.
(1) Maybe you have two algorithms (a state-of-the-art one and a new one you are proposing/validating), and you want to compare both. In that case, things are simple and you can just perform the procedure outlined by Michele and apply standard statistical tests to see whether you observe a statistical significant difference in performance.
(2) Maybe you are more interested in finding the best algorithm in a large pool of algorithms (and parameter setting of those), because you are interested in building a good model for a particular application. Then, keep in mind the following two points:
(a) when evaluating the performance of an algorithm (and parameter setting) on a dataset, the accuracy of the performance measurement depends on the size of your test set. Moreover, the more algorithms you compare, the higher is the probability that there are algorithms which look better than they are on average on the population from which the dataset is drawn. So it is not always possible to confidently find the "best" algorithm (parameter setting).
(b) given that anyway you can't find the "ideal" algorithm using a limited dataset, you may prefer to explore only part of the space of possible algorithms/parameters (e.g. do some gradient descent on parameters) and reduce computation time.
Trying a tool like WEKA:
http://www.cs.waikato.ac.nz/ml/weka/
is the best way to decide which algorithm fits to the problem at hand.
The algorithm that provides the best solution for the same training set and within a certain period of time is the best algorithm. In most problems are required to provide an optimal solution within a specific time.
Generally Machine Learning algorithms are evaluated on certain performance indicators. For example, metrics like Accuracy, Precision, Recall, F-Measure, NMI, Rand Index etc are used in application Machine Learning Algorithms to quite a few domains including Text Analytics.
- Badges
- Science topic