Parallel Processing is one of the techniques to improve the performance and reduce the running time of batch jobs significantly.

I had once a requirement where a series of application engines running on daily basis were taking around 24 hrs. These application engine used to process 50 million rows. The longer run time was certainly impacting the client. So, we started analyzing , found the bottlenecks and looked for solutions.

From SQL tuning, Set based processing to Parallel Processing we ended up reducing the run time of whole series of batch jobs 8 hrs..a reduction of 16 hrs…!!!

Parallel Processing was the key to achieve it.

So, how do we implement it. It’s easy , just two steps.

1. Logically divide the whole data so that you can trigger multiple instances of same application engine, and each instances processes the data aligned to it.

2. Design your application engine in such a way that each instance processes the data allocated to it.

There should not be any overlapping of data between the different instances or you will end up processing the same data by multiple instances.