It is my personal opinion, I got success with beowulf cluster for multi-GPU cluster. So my suggestion is:

You can build that using beowulf cluster.

Also, you can try torque server to submit jobs (fourth link)

https://www-users.cs.york.ac.uk/~mjf/pi_cluster/src/Building_a_simple_Beowulf_cluster.html

https://devblogs.nvidia.com/parallelforall/how-build-gpu-accelerated-research-cluster/

https://hemprasad.wordpress.com/2014/12/22/how-to-build-a-gpu-accelerated-cluster/

http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/3-nodes/NVIDIAGPGPUs.htm

Frankly said, we can learn to built a small cluster using some sets of raspberry phi units (http://makezine.com/projects/build-a-compact-4-node-raspberry-pi-cluster/).

In order to run the model in a parallel computing environment, I am afraid that you should modify the software. It should be compatible to be run parallel. Especially related to its input output mechanism.

If we become confident enough, we can start to think about arranging a bigger cluster (http://www.wikihow.com/Build-a-Supercomputer and http://proj1.sinica.edu.tw/~statphys/computer/buildPara.html).

Hope this helps. Good luck.

 Hi Panchatcharam Mariappan and Santosa Sandy Putra, 

Thanks for your kind responses.   I'll give it a try later. 

Regards.

Hi,

I would recommend Rocks for building and maintaining a cluster. For queuing system you can use Slurm. See the links below.

https://slurm.schedmd.com/

http://www.rocksclusters.org/

How you build your cluster depends on what you want to do with it.

Do you already have software? What are the requirements of the software?

If you intend to use CUDA or similar libraries with GPUs (as opposed to CPUs) you should refer to the recommendations in the Nvidia documentation. Not all systems are created equally.

Another aspect is that, yes people are using RaspberryPi for clustering, **However**, what are your doing? Is it just an experiment to try clustering, or is it a production cluster for doing actual work?

Beowulf does not automatically use your GPU cores without using proper librarys. Same with the others. There are also a differences between MPICH, OpenMP, and GPU programming that you should research before spending your money. Those are all different things that can be used to develop parallel programs, but you need to understand where to use them to make them effective.

From my experience it's better to start with a clear plan for your software implementation and your goals. Decide what libraries you are using, etc..

Boost libraries are very good, but like all libraries you will probably need to write some other code to fully support your GPUs.

Dear Peter Willis,

Thanks for your response. Maybe I should clarify my problem: 

We use Nvidia GPUs (GTX1070) to do some scientific calculation, which are distributed in several separated machines. The software is called VASP. But I am not clear about "what is the requirements of the software" with regard to the set-up of GPU clusters.

Thanks for your help. 

Regards!

Looks like you just need to set up your CUDA directory in the Makefile. Check out the link.

http://cms.mpi.univie.ac.at/wiki/index.php/Installing_VASP#For_the_GPU_port

We have successfully compiled vasp_gpu, and it runs well. But we want to combine the separated GPU units and build a job queuing system for all the GPUs. I know "PBS_ROCK" works for CPU. Whether there is a similar system serving GPU？ 

If each of your VASP processes will be running independently of one another (if there is no need for them to interact / communicate), then you may want to look at Swift as a language for launching those instances:  

  http://swift-lang.org/main/

Note that this 'Swift' is completely different from Apple's Swift programming language.

If your VASP processes need to interact with one another, then you probably want to use the message passing interface (MPI) to launch and enable communication between them.  There are two free / open-source versions of MPI:

 OpenMPI: https://www.open-mpi.org

 MPICH: https://www.mpich.org

Good luck!

MPI should happen regardless of the GPU compilation status, as long as the application is MPI capable. They are two separate things.

The MPI settings are also in the Makefile, although there is no specific flag that indicates the program should compile via MPI (mpicc). Unless the presence of that variable constitutes the flag.

Quote:

"MPI_INC

Path to MPI include files so the CUDA compiler can find them. For example:

MPI_INC := /opt/openmpi/include

These can often be found with mpicc --show."

Logically, providing a variable for MPI as a part of the build would indicate the software is probably MPI capable.

Do you have MPI development files installed? You would need headers and binaries.

What is your compiler environment? (gcc, Intel, Portland)  The compiler may be expecting the headers to be in a specific location or an environment variable to be set.

If you are running on a Linux distro, there should be MPI dev packages available via your package manager. That would save the need to compile from source. Compiling MPI from source can be a bit better because the MPI build adjusts for the machine(s) you are running.

I suggest SLURM for job control. There are several front ends for that. Here is one:

https://slurm.schedmd.com/overview.html

 Dear Prof. Adams and Dear Peter, 

Thanks for your advice. We have MPI and the compiler is gcc. Maybe it is my mistake and a CPU job queuing system is enough. 

Regards and Merry Christmas!

Cheng. 

Dear Cheng,

In any case you should install a GPU-aware job manager. To my knowledge SLURM does the job well (I have been using it for a while now).

Have fun building the cluster!

Dear Mr. Cheng,

Hello! I am just starting to build a production cluster with a similar specification with yours (multiple node with gtx 1070 each). My professor have these hardware and wants me to explore gpu-accelerated VASP (one of the primary code/programs used in the lab) and so here I am. May I ask you a few questions regarding your finalized build?

1. What cluster software (name and version) did you use ? In my research, it seems that ROCKS is good and particularly user friendly (in terms of installation).

2. What MPI and job manager did you use? Currently, I am still exploring the options for this.

3. What compiler did you use for the vasp (with gpu support)? I think I'm leaning towards the free gcc compiler.

4. Did you have any difficulty in any particular step (bug or incompatibility wise)?

Thank you for your time.

Hi Kurt,

Sorry for the delayed reply. At the moment, I am not using GPU clusters, but I will try my best to answer your questions.

1. What cluster software (name and version) did you use ? In my research, it seems that ROCKS is good and particularly user friendly (in terms of installation)

As you said, ROCKS is really good, and we used it for our cpu cluster.

2. What MPI and job manager did you use? Currently, I am still exploring the options for this.

OPEN MPI.

3. What compiler did you use for the vasp (with gpu support)? I think I'm leaning towards the free gcc compiler.

I think it is gcc.

4. Did you have any difficulty in any particular step (bug or incompatibility wise)?

Actually we did not build up a GPU cluster in the end because we found that we did not need a GPU cluster, and a CPU cluster was good enough for our group.

Hope it helps.

Best