I have written a python code to carry out genetic algorithm optimization, but it is too slow. I would like to know how to run the same in parallel ?
For more clarity, another python code will be called by my code for, say 100 times, I wanted to divide this into 4 CPUs. So that 25 times the outside python code is solved by each CPU. Thereby increasing the speed.
Thanks in advance!
I design, administer, support and use large HPC clusters. I was curious about Python for HPC, so did some basic MPI tests. The results were shockingly bad - I can probably dig them up or repeat them if there is interest.
The point is that if you invest in a cluster with decent networking (that is, Infiniband), you are paying a great deal for 1 uS message-passing, and it's a crying shame if merely wrapping the MPI in Python degrades performance by several times. On the other hand, if you have only a gigabit fabric, your messaging latency is going to be in the 30-50 uS range, so Python's latency is a relative non-issue. (Anything done on virtualized infrastructure is also usually high-latency, and even 10Gb doesn't help that much.) If your code is primarily bandwidth-sensitive, I would not expect a big problem from Python, at least on modern hardware (ie, even Python's poor object layout, indirection and GC can't burn all of the ~60 GB/s that a dual-socket haswell provides!)
Naturally, if your application is embarrassingly parallel, it doesn't really matter how efficient your communication is. (That's almost the *definition* of EP!)
The GIL (or global interpreter lock) is not something that cannot be worked around, and in fact parallelization in Python is quite simple to implement to implement in Python depending on the specifics of your coding structure. Be aware of the GIL, even though you parallelize your code you may see no increases in run-time if the GIL is not worked around.
Look up map_async under "Process Pools" here:
https://docs.python.org/2.7/library/multiprocessing.html
Python 3 is my expertise, and in 3.3+, starmap_async is what I utilize for my GAs.
Additionally, take a look at Cython. Through it, there are many ways to parallelize as well. http://cython.org/ Cython allows you to completely work around the GIL.
I have using for a while the mpi4py library. Works like a charme. See documentation at:
http://mpi4py.scipy.org/docs/mpi4py.pdf
It is simple and efficient. If you give me more details I may help you. What type of genetic optimization are you performing?
I would suggest you to simply go through the below link then try for Prof João Carlos Henriques ans.
https://computing.llnl.gov/tutorials/parallel_comp/
this link will help you think parallel
@Henning, thanks for the update.
Ya, they are the part of same machine.
I can warmly recommend IPython for parallel computing:
http://ipython.org/ipython-doc/dev/parallel/
It takes a lot of the pain out of parallelisation with python.
Actually, I can recommend IPython for basically everything that you might want to do with python ;) Just look at IPython notebooks for fully reproducible interactive computing.
I design, administer, support and use large HPC clusters. I was curious about Python for HPC, so did some basic MPI tests. The results were shockingly bad - I can probably dig them up or repeat them if there is interest.
The point is that if you invest in a cluster with decent networking (that is, Infiniband), you are paying a great deal for 1 uS message-passing, and it's a crying shame if merely wrapping the MPI in Python degrades performance by several times. On the other hand, if you have only a gigabit fabric, your messaging latency is going to be in the 30-50 uS range, so Python's latency is a relative non-issue. (Anything done on virtualized infrastructure is also usually high-latency, and even 10Gb doesn't help that much.) If your code is primarily bandwidth-sensitive, I would not expect a big problem from Python, at least on modern hardware (ie, even Python's poor object layout, indirection and GC can't burn all of the ~60 GB/s that a dual-socket haswell provides!)
Naturally, if your application is embarrassingly parallel, it doesn't really matter how efficient your communication is. (That's almost the *definition* of EP!)
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python