I want to develop one program in hadoop and the same in either opencl or CUDA programming language. I want to know can we compare both of them accroding to their performance, by running one on a GPU and the other one on equivalent number of machines?
I am not exactly sure how you want to make the results comparable. You can change the results either way so that either HADOOP or your GPGPU implementation wins. If HADOOP is slower, add more CPUs. If you pick a different GPU, again your results will change. What will be the basis to compare different architectures? Should they have the same number of flops? In that case HADOOP might be slower due to network overhead. So, how are you accommodating for this in the results? Using the same number of cores is not a fair comparison because CPUs have about 2 times higher clock speed compared to GPUs.
So, how would you define number of machines equivalent to GPU?
Furthermore, if you take the exact same algorithm for HADOOP and GPGPU it will be better optimized for one of the two. Now, if you start to optimize the two algorithms separately, each for its architecture, the one that is better optimized will win. So, in my opinion, even if you manage to provide an equivalence for the number of machines, the only statement you can make is which algorithm you were able to better optimize.
I don't want to offend you, but I want to make clear that it is really hard to find objective criteria.
Maybe, a more interesting question is to find the break-even point between the GPU and the number of cores for HADOOP for your implementations. Then consider the amount of work it takes to optimize for the two architectures. Also, at the break-even point which hardware would be the cheapest? If you can check the GPGPU implementation on different GPUs this could help to support these questions. In the end you will have an answer for decision makers: For these kinds of algorithms which would be the better way to go if you want best performance? GPGPU or HADOOP? (Finally, this depends on your problem size, the amount of money you want to spend, and the time you want to wait for results.)
Hello Chirag,
I've being working with Hadoop and OpenMP/MPI, and our model has um process that can be used with CUDA, but i never used it.
If what you wants to achieve (in terms of processing and results) can be done in a HADOOP context, than yes, you can compare the performance, i.e., the time that takes to complete the processing on each environment.Remember that HADOOP is meant to be used with DATA parallelization, where you launchs X instances of a program and each instance (usually) works on a different set of data.
While HADOOP can provide great flexibility / scalability (one of its main advantages), I expect that a CUDA/OpenCL program, if well optimized, to be much faster in the "same " conditions (read number of cores here).Cheers
Eduardo Jauch.
Neighter opencl nor CUDA are programming languages. They're just frameworks, a library set.
Obviously you can make this kind of comparison, having on mind that every solution depending on it's architecture - even OS - may require different optimization approach (even on the level of chosen algorithms) to obtain highest possible performance.
Take a look at the following paper. The authors in that paper compared several different implementations on various platforms.
https://www.researchgate.net/profile/David_Ediger/publication/262159283_A_performance_evaluation_of_open_source_graph_databases/links/54ee1cc30cf2e28308641e06.pdf
Article A Brief Study of Open Source Graph Databases
Conference Paper A performance evaluation of open source graph databases
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages