Hello - I need to apply multi-threading to a subroutine in python on spark. Any feedback or references w.r.t. getting a fast and efficient outcome would be greatly appreciated. FYI the subroutine involves the creation of a sparse matrix from the input data followed by a logical row reduction within a while do-loop. Pyspark has specific modules but I need to create one. Any feedback on this would be greatly appreciated. On-line docs do not help very much. This is specialized knowledge. I don't want to have to worry about e.g. issues of shared memory that can negatively affect performance. Thank you. Best - Tony Scott
Hi,
First of all, If you are using a single machine, I would highly recommend using threading in Python or some other way to use all the cores at the same time.
If you have a while-do loop, I would recommend trying to parallelize it with some of Spark's mapping routines, since logical cycles (for, while) are not very efficient in pySpark.
Also, I would only recommend Spark if you have multiple machines, Spark in a single machine is hardly more efficient than a good Python implementation.
Is spark a requirement?
As Leon commented, threading in Python is the best option for a single machine with many cores.
If you have several machines the best option is MPI for python:
http://mpi4py.scipy.org/docs/
And you can also combine MPI with multhreading to use several machines and all the cores in every machine.
Spark offer fault tolerance and things like that but is very slow (compared with mpi of multithreading).
Anyway, if you must use Spark I think we need more information about you process to help you.
Thank you Leon and Roberto. Yes, Spark is a requirement. What needs parallelization is within a for loop but what is inside that loop is somewhat complicated code with the creation of a sparse matrix followed by sums over columns and a final logical row reduction within a while-do loop. The final result is then represented by a much reduced sparse matrix. After the loop all the resulting sparse matrices are concatenated and row reduced one final time. It's a bit like a map-reduce operation. I have looked at "parfor" in MATLAB (joblib in python) but got only moderate results (2 processors only). The spark machine I am using offers more in terms of cores and threads though more complicated. Thanks again.
I am very disappointed! The idea of parallelization was to get improved computational performance. Neither threading nor multiprocessing in python yield faster results than a simple list comprehension which in general does not produce a faster result than a for-loop unless the function is VERY simple. In part, it's because of the Global Interpreter Lock (GIL). David Beazly showed a seminar about this point. Shared memory is often a liability. I had hoped that the Java layer in pyspark would produce better results but it doesn't. A machine with 48 cpu cores using pyspark produces results no faster than what I can get on my MAC with 4 cpu cores. It's a disgrace. IMHO, it's a design flaw.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python