The most straightforward way is to use the multiprocessing module. It creates an object that is a que manager. It has methods such as .map() and .map_async(), which, as arguments, take a function and an iterable of function inputs. The manager then creates worker threads and doles out function instances to them. It's automatic enough to figure out how many CPU cores you have, and creates a separate process for each one.

This is very primitive multiprocesing - workers can't talk to each other. But if you simply need to crunch through a bunch of data using identical processing, where processing of each batch does not depend on results of other batches, then it's perfect.

Thanks Boris !! will try to implement it in my script. 

Hi Ashish,

I have tried multi-process for doing some tasks which each process create a result file. Then the program detects the amount of the not null result files as a signal to start to assemble them. Hope this helps. Let me know if you'd like the code.

Best,

Tianrun

Thanks Tianrun. It seems my problem is more due to file I/O operations rather than computations. Multi threading did not help much in my case. 

import multiprocessing as mp

#

# create pool with amount of cores

pool = mp.Pool(processes=4)

#

# run processes in pool

pool.map(some_function, list_of_arguments_for_some_function)