I have encountered an odd problem.
I am trying to run parallel Fluent on a cluster include 7 nodes and 80 cores. I am using a Gigabit Switch and cat6 cables.
While I am trying to read case and data, transfer rate on network about 20 mbps but after a few seconds it drops to 200 bytes per second and it is excruciating,case implemented completely and data take unlimited time.
I should mention that all windows HPC cluster tests done completely.
thanks in advance.
We run Linux clusters here, so I do not have the exact commands for Windows HPC clustering, however, some basic troubleshooting should be the same.
The first three most likely issuesm which you can probably check as a user.
1) Check fro memory thrashing:
You can check that the Fluent processes do launch on separate nodesin the cluster, and that they are not consuming so much memory as to cause the application to page/swap. If that happens, all of the I/O being done is to swap and none for solving your problem. I believe Fluent uses the cnf= argument to specify the file with a list of the nodes to launch on. Make sure that you are using the nodes assigned to you by whatever scheduler you are using. This is ${PBS_NODEFILE} for Torque or PBS.
2) Check that no programs other than yours are running on those nodes. Leftover orphaned processes can also cause memory thrashing. (ps -aef in a pbsdsh script would do this in a Linux cluster running Torque, don't know for Windows)
3) Check the network with some simple file transfers to make sure there is not a simple network problem.
If the above check out, and you are the admin, then I'd check that the physical network components and the network configuration is correct. You could have a bad cable, a bad network port, a loop in your network, ...
Dear james
many thanks!
I may have to make a confession,
unfortunately I was running on 5 nodes but adding a node cause this problem.
I wonder that exactly case and data volume will be transferred but reading data last
and don't culminate.
thank you for your fruitful answers.
I am avidly waiting for any other idea.
Mahdi,
I would check the new node, and maybe try 5 nodes, excluding one of the 5 sand including the new node.
If the new node has less memory and is thrashing, then this may slow the whole fluent job down. You might also want to run a MEMCHECK to determine if you have memory problems.
- Badges
- Science topic