It is common practice that a newly proposed method is compared with previous methods but there are many factors seem not to be taken into consideration such as the different machines on which the algorithms are run. Is there any "proper" way of comparing algorithms running on different machines?
Not "different machines" (that is vague) but different operating systems, different versions of the same operating systems, or different programming languages and different versions of the same programming language, or parallel vs. nonparallel computation, or different platforms, etc. These can all be compared, yes, but you need to ask yourself: is it really worth the time to do all that optimization before obtaining the computational results that you require.
If you would like to compare the new method with previous methods, I suggest you start with theoretical analysis before actual implementation. Computing space and time complexity functions (possibly represented by asymptotic notations if only rough comparison is needed) will give you a good idea about which method is faster and/or requires less memory. In addition, consider computing the threshold where increasing the size of input starts to make one method out-perform the other.
If your new method survives the theoretical analysis with promising performance, you can start implementation by choosing a platform and conditions that resemble the best previous method you can find (if that is reasonably doable). Finally, you can compare the implemented methods based on different factors, as mentioned by Mr. Vicente Mendoza Reyes.
I agree that a theoretical comparison would be the first thing to do. However an empirical comparison is probably more telling, specially in case when the different algorithms have a similar complexity.
For this empirical comparison I would not advice to recode all algorithms in the same framework to make them "comparable". A specific implementation of a given algorithm might be best suited for a specific framework/language, then doing this change will already bias the comparison against that algorithm.
In my opinion you should use an operation that is key for all algorithms, for instance the computation of entropy, and compare how many computations each algorithm needs, in relation to the solution given by each if not all produce the same output.
Typically in order to effectively compare 2 different algorithms you should aim to remove all differences between the environment they run in, meaning both hardware and programming languages.
In order to do this the main approach is to use something like big O notation, and compare the bumber of operations to perform the steps. Big O notation is used to compare the number of steps based on the number of inputs and creates a baseline to compare against all other algorithms. Even if you do not use Big O for this case i highly recommend you learn it.
For example O (n) is better then O (n^2) where n is the amount of input. E.g. 1000 items to sort...
By comparing algorithms this way the actual implementation is ignored and the performance of the algorithm is not hindered by problems in the specific machine or the programming language.
Whenever you publish any work formally and if the work relates to proving any particular algorithm better over anything, you describe your assumptions and sometimes the environment that has been used to conduct experiments. For example, my previous year project was to automate Virtual Machine management in clouds using reinforcement learning algorithms I used Q-learning and SARSA algorithm for implementation of the MDP based system. I compared these two algorithms and mentioned which cloud environment and OS I will be using. It will be way too clumsy to consider the factors that you have mentioned. So, if anyone wants to test it in a different platform with variant machine parameter, let him do that sir...
As Vicente Mendoza says, the results reflected in the literature are being obtained by other authors helped by different operating systems, different versions of the same operating systems, different programming languages, etc. with regards of the one obtained by the new author. By this the crteria for comparision must to be not machine time but other criteria, I prefer the quantity of iterations for obtain the same parametr, by example quantiy of calculation of the objective function during the optimization processes, etc
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Algorithms