We are doing a benchmark of an urban case with 2 nested domains, resolutions 36 km and 12 km, and comparing the results it seems that running with the same set up it takes almost twice time using 128 processor via parallel machines instead of 64 in a single machine. Does anyone know this difference? Could be some issue with the efficiency of the communication between processors?
How you connect machines? what type of network you have? Maybe the speed of network is delaying calculations.
Have you checked network latency, bandwidth and the load on the network?
Fabio,
The answer to your question is dependent on your hardware and software setup on these computer systems. Processing speed will inherently be bounded by the slowest link in your network. Below are some likely causes of this phenomena:
1. A single computer running multiple CPU cores will ALWAYS be faster than multiple networked computers running the same exact CPUs.
On a single machine cores will communicate through the motherboard bus and communication between cores will be as fast as the local bus speed. A network setup the CPUs now need to communicate between the bus, the network adapter, on the local network, back to the second network adapter, bus, then CPU.
2. The software libraries used to communicate through the network and bus may not be optimized.
When you compile and execute your program on these two systems the compiler will have to link into the network libraries, which may not have been optimized correctly.
3. The shared filesystem will ALWAYS be slower than memory
Running on a cluster system will inherantly require writing to a shared filesystem disk between processors. This system will also be slower than using disk/memory on a single computer.
In summary, your speed is determined by your 'weakest link'. You WILL have a slower speed on a networked cluster with more higher clock CPU's than a single machine. However, your capabilities increase substantially the larger your program becomes. If you only need to run small runs on a single machine then stay with that, but once your program becomes too great for a single machine then you need to step up to a parallel system.
Thank you all so much, we really appreciate your answers. We'll check all the information above to find out the problem.
on computational cost memory latency and bandwith occupy the most relevant part. I agree with the previous suggestions, another point is the memory grain of computaion and memory hierarchy on single processor and complessive memory hierarchy on a multinode computer. Man memory optimization have been impemented during the past on chip according the run-time system. Try to use profiling tools to achieve information on single process registry and memory use and tools (such as mpi profile if you use MPI library) to collect performance indicator on pararallel execution. In general the grain of parallelism should drive the choise between DMS and Shared systems
Another thing to consider is the size of your domains. Make sure you have enough computational load when you partition the mesh into 128 processors. On high end systems with good interconnects, we have noticed that the scalability of WRF degrades as the computation component starts to decrease and the communication becomes a larger fraction of the total time.
I think it is due to compiling WRF, serial, smpar, parallel or dmpar
Querido Fabio,
many people pointed out reasons why WRF may be slower when executed on 128 procs on a distributed machined than when executed on a 64 procs on a single machine. There are lot of issues like disk performance, data distribution (i.e. load balancing), communication issues (think on differences using network or memory to transfer data), the processor themselves (they aren't the same from your words, I guess), which translates into different IPC, among many other issues.
If you want to investigate further, I'd encourage you using performance tools with WRF on your systems. There are many performance tools out there that can help you such as Paraver [1], Scalasca [2], and/or TAU [3]. I'm pretty positive that using them will help you understanding why the difference between those executions.
Best.
[1] http://www.bsc.es/computer-sciences/performance-tools/examples-analysis
[2] http://www.scalasca.org/
[3] http://www.cs.uoregon.edu/research/tau/home.php
One quick check just to be sure:
Check your largest data partition and compare it with the cache size. It is possible that the slowdown is due to cache page faults. (if the data cannot fit into processor cache, it will result in page faults = multiple times slower running time).
Try halving the data and testing it again. if you get similar results it is not due to page faults. Otherwise, think of a more considerate partitioning sceme
You can see this: http://weather.arsc.edu/Events/AWS10/Presentations/MortonBenchmark.pdf
My best guess is that small domains, my assumption, do not scale well with many cores. You start loosing performance (scalability) when your tiles/core have < 100 grid horizontal points.
- Badges
- Science topic
- Similar topics
- Geoscience
- Atmospheric Sciences
- Climate Science
- Climate Change