Hi,
I know that some of Support Vector Machine approaches and other machine learning approaches use the methodology of reducing the number of sample from the training set to reduce the computational run-time. However, this method can work very well on large training sets if they nearly have instances characteristics that can represent with the small portions (small samples) of these training sets. However, it will not do the same outperformance on training sets that have a lot of variations in the instances.
Please, is there any method in machine learning methods to reduce the computational run-time with considering all sample to be involved in the learning approach?
Thanks
Osman
Recently there have been proposed several accelerated methods for finite-sum minimization (SVM optimization problem can be stated as such).
https://arxiv.org/pdf/1409.3257v2.pdf
http://pwp.gatech.edu/guanghui-lan/wp-content/uploads/sites/330/2016/02/OptRandom10-18.pdf
https://arxiv.org/pdf/1309.2375v2.pdf
https://arxiv.org/abs/1603.05953
These methods are very fast advanced versions of stochastic gradient descent.
Also please refer to non-acceleratad methods for finite-sum minimization. The references can be found in the papers I listed above.
Some specific algorithms for SVMs or other methods will run faster than other algorithms. You need to read the literature to find faster algorithms.
Sampling to reduce the number of records can be problematic. First, once sample size becomes too small, the training will not work properly. Second, as you noted, the sample (or the original set of records) may not be representative of new data. Training with unrepresentative data will (almost) never work.
The fastest one is Linear Regression and then ES-Rank (my approach) in large datasets that consider all training samples in the learning procedure. However, ES-Rank is in the third position on running time compared to the other 14 machine learning methods but most of them did not consider all training samples in the learning procedure. I made a comparison between 14 machine learning methods in IR with my method in SAC 2017 paper. However, Linear Regression is not efficient enough comparing with the rest of methods. My method and Linear Regression include all training samples on the learning procedures.
1. you can look at techniques to parallelize the learning process, and run it on computing clusters or GPUs
2. some techniques such as batch stochastic gradient descent look at smaller subsets of the dataset at a time, and do this iteratively to reach a suboptimal solution (if you stop the process after a fixed time)
I made the fastest Evolutionary Gradient Strategy approach in IR that consider all training instances in the learning procedure that is disseminated from ES-Rank and it is under the procedure of publishing the research outcomes
Recently there have been proposed several accelerated methods for finite-sum minimization (SVM optimization problem can be stated as such).
https://arxiv.org/pdf/1409.3257v2.pdf
http://pwp.gatech.edu/guanghui-lan/wp-content/uploads/sites/330/2016/02/OptRandom10-18.pdf
https://arxiv.org/pdf/1309.2375v2.pdf
https://arxiv.org/abs/1603.05953
These methods are very fast advanced versions of stochastic gradient descent.
Also please refer to non-acceleratad methods for finite-sum minimization. The references can be found in the papers I listed above.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining