I am trying to run a non-linear finite element MATLAB code whose basic structure is as shown below. It has an outer for-loop and inside it, there are two parfor loops.
for i = 1 : total_load_steps
parfor w = 1 : number_of_elements
//Elemental matrices and vectors computation
end
// Sparse assembly and solve
parfor w = 1:ele
// update variables
end
end
The main script described above and necessary functions present inside it are stored in my local computer.
The 'userpath' command is set to the directory on my local computer from which I am running MATLAB.
The HPC cluster is called from my local machine using 'parcluster' command and the MATLAB script named 'My_model' is submitted to the HPC from my local computer. The following commands are used in the process:
configCluster;
// Entire my user-name and password //
cp = parcluster;
cp.AdditionalProperties.QueueName = 'highmemory';
cp.AdditionalProperties.WallTime = '10:00:00';
cp.AdditionalProperties.ProcsPerNode = 48;
cp.saveProfile
job_1 = cp.batch('My_model', 'Pool', 96, 'CurrentFolder' ,'.');
Strangely, even after assigning 96 cores in HPC, the computation runs slower than my local computer which uses 10 cores only. Moreover, when I increase the number of cores on the HPC, the computation runs even more slowly.
I am unable to find out the exact reason behind such an undesirable behaviour. Could anyone provide any insight as to why is this happening and how can I resolve it to speed-up the computation in the HPC ?
There could be several reasons why your MATLAB computation using a parallel cluster on an HPC is slower than your local computer. Some possible factors include:
To address these issues and improve your code's performance on the HPC cluster, consider the following:
Alessandro Rizzo Thank you sir for providing your valuable insights into this problem.
I believe that this problem in my code is caused by 'communication overhead' about which you have discussed in point no. 1.
In my parallel for loop, I have about 35000 numbers of independent computations to be performed and these are distributed over the 96 cores which I am assigning in the HPC. For each of those independent computations, I need to transfer some variables into the cores. The communication of these variables to the cores and communicating back the result variables from each core is probably increasing the total computation-time
Could you please provide me any knowledge or resource using which I can optimize the data-transfer between nodes?
Putting data used by processes of a given core in the same memory as accessed by that core is a very good idea.
Why not start with one core? Measure the speed? Then expand cores until a runtime asymptote is reached. For that model size, that would be a good estimate of future runtimes.
Certainly, Sandipan Baruah! I'm glad the information provided earlier was helpful. To optimize data transfer between nodes and reduce communication overhead in your MATLAB code, consider the following strategies:
I hope these suggestions and resources help you in optimizing data-transfer between nodes and improving the performance of your MATLAB code on the HPC cluster. Good luck!
Hi Sandipan Baruah
There are several possible reasons why your code runs slower on the HPC cluster than on your local computer, even with more cores. Here are some factors that you should consider:
- Parallel overhead. There is overhead in calling parfor instead of for. If function evaluations are fast, this overhead could become appreciable. In particular, solving a problem in parallel can be slower than solving the problem serially1.
- Passing parameters. Parameters are automatically passed to worker machines during the execution of parallel computations. If there are a large number of parameters, or they take a large amount of memory, passing them may slow the execution of your computation1.
- Contention for resources: network and computing. If the network of worker machines has low bandwidth or high latency, computation could be slowed. Factors That Affect Results Some factors may affect numerical results when using parallel processing1.
- No nested parfor loops. This is described in Nested Parallel Functions . parfor does not work in parallel when called from within another parfor loop. If you have programmed your objective or constraint functions to take advantage of parallel processing, the limitation of no nested parfor loops may cause a solver to run more slowly than you expect1. When executing serially, parfor loops run slower than for loops. Therefore, for best performance, ensure that only your outermost parallel loop calls parfor1.
- Where to create arrays. When you create a large array in the client before your parfor-loop, and access it within the loop, you might observe slow execution of your code. To improve performance, tell each MATLAB ® worker to create its own arrays, or portions of them, in parallel. You can save the time of transferring data from client to workers by asking each worker to create its own copy of these arrays, in parallel, inside the loop2.
To resolve these issues, you can try some of the following suggestions:
- Use tic and toc to measure the elapsed time of your code segments and identify the bottlenecks.
- Use ticBytes and tocBytes to measure how much data is transferred to and from the workers in the parallel pool2.
- Use parallel.pool.Constant to establish variables on the pool workers before the loop. These variables remain on the workers after the loop finishes, and remain available for multiple parfor-loops. You might improve performance using parallel.pool.Constant, because the data is transferred only once to the workers2.
- Avoid nested parfor loops and use only one level of parallelization.
- Create arrays inside the parfor loop instead of before it, or use slicing to access only the relevant portions of the arrays2.
I hope this helps you improve your code performance on the HPC cluster. For more information and examples, you can refer to these web pages: Improve parfor Performance, Improving Performance with Parallel Computing, parfor loop for loop and batch processing in Matlab, A concrete example of a parfor loop in Matlab that outperforms the for loop.
Refrences:
https://www.mathworks.com/help/optim/ug/improving-performance-with-parallel-computing.html
https://www.mathworks.com/help/parallel-computing/improve-parfor-performance.html
https://www.mathworks.com/help/parallel-computing/improve-parfor-performance.html
https://www.mathworks.com/help/optim/ug/improving-performance-with-parallel-computing.html
https://stackoverflow.com/questions/23898126/parfor-loop-for-loop-and-batch-processing-in-matlab
https://stackoverflow.com/questions/11934910/a-concrete-example-of-a-parfor-loop-in-matlab-that-outperforms-the-for-loop
- Badges
- Science topic
- Similar topics
- Correction