There are some papers that compare accuracies versus training time, but usually it is not very relevant for two main reasons:

That being said, there are some context where the training time is important and it's usually related to streaming data. In that case, you need a training algorithm that can work online and then the relevant metric is the speed at which it can process examples versus the accuracy.

Another topic where training time is taken into account is when you have resources constraints (cpu, memory). In that case since you cannot use computational tricks the time of regular algorithms (e.g., kernel ridge regression) can be greatly increased and computation times are often included.

Hi David,

I think if you increase the number of bagging or number of leaves or neurones or changing learning rate or changing regularisation rate in most of the machine learning techniques, it will increase accuracy but will increase the computational runtime. The same issue if you increase the number of iterations in some computational intelligence techniques, it will increase the accuracy against evolving time. In other cases, machine learning techniques in some parameter settings may be underfitting or overfitting accuracy that causes the need for packages become available with detailed parameter settings for every paper in machine learning and computational intelligence research. 

Thanks,

Osman

However, there are little papers that may include these things together (no more 1% of machine learning and computational intelligence papers that include computational runtime comparison vs accuracy, detailed parameter settings and library package )

Thanks,

Osman

Hi Osman,

While I agree training time is relevant at some level of application in most cases it is very hard to compare it. Training time on what hardware and with what application of the algorithm? 

Another problem is would you consider training time of the final model or for the whole model search? architecture and hyper parameter search? What if one performs some feature engineering, should the training time account for it.

Here's a little example from personal experience:

goal was to predict a post-translational event based on primary sequence of it surroundings.

I spent weeks testing various predictors derived from primary sequence and their combinations, for each finding a set of decent hyper parameters. And when I found some good combinations, the final model train time was under a minute. What should I report? There is no consensus. 

Regards,

Milan

At a macro level the various competitions that invite researchers to submit machine learning solutions for a common problem address the effective comparison of different methods. With each competition, it is hoped that there is enough variety of techniques that a valid enough comparison has been made. Of course, it may be that only popular, quick or easy techniques are used within the various solutions but any other approach requires a research team to spend a very long time trying every possible combination of meta parameter, technique and data feature.  While competitions don't directly address your question about computation time, in a sense they do answer it indirectly in that the various submissions have addressed the same problem within the same timeframe.

Computation time is a factor in my own work and what I have learnt to do is to test ideas using smaller models and then once I think that I have found a reasonable set of circumstances then (sometimes) I scale up, looking for a more optimal solution. However, mostly I am disinterested in proving that one technique is the "best" and more interested in showing that a particular technique works well enough to be usable. This, however, reflects my own research aims and is not intended to be a criticism of researchers who do want to try to prove that their technique is better than someone else's.

In terms of the other part of your question that asks for pointers to packages that can easily be used to implement various machine learning algorithms, the two environments that I would point to are WEKA and the R language. Both include a wide variety of easily implemented algorithms but both are not complete. I use R but I know of colleagues who use WEKA. It is also worth noting that neither environment can claim that the implementation of all algorithms within the environment is the most computationally efficient implementation that is possible.

Computation time is relevant if you are implementing a particular technique in real life and is also relevant when doing a large number of differing runs to try to optimise meta parameters but is not a good measure in isolation. In a recent situation I ran two different implementations of the same ML technique. One implementation was ported to R from MATLAB while the other was written specifically for R.  The MATLAB ported code included one additional feature that was not present in the written for R code. The MATLAB ported code was at least 10 times less efficient than the written for R code but produced slightly better results (as a result of the additional feature). How should I compare these two situations if I put a lot of emphasis on computational efficiency?

If you want to use computational efficiency as an effective measure then firstly you must ensure that each of your situations that you are comparing are implemented as efficiently as possible and then you have the dilemma of time (wasted) spent trying both to implement the solution as efficiently as possible and proving that you have done so.

Of course, some solutions can be shown to be more efficient at the conceptual level but failing this, efficiency is as much an implementation issue as it is a algorithmic issue.

Hi Doug,

TIME is money in some cases. If the machine learning algorithm consumes too much runtime, this will consume effort and resources allocated for the machine learning algorithm and more time to have good results. Also, no one can said machine learning algorithm outperformed other algorithms in terms of accuracy without determine fixed settings for each algorithms. Sometimes the same machine learning algorithm can outperformed itself in various settings. i.e. various settings for a machine learning technique can cause a various performances and various accuracy values.

Thanks

Osman

Hi Osman,

I also do agree with you on the aspect you mentioned about detail and convincing explanation is really not captured in many papers. I even thought if these experiments are even carried out as they claimed. And i think this is usually common in the so called relevant papers.

Meanwhile, I agree with the fact that computational time is a vital measure for machine learning comparism but not really clearly defined in most cases.