I think I need a lot more detail. Apologies for the "random thoughts" covering the various bases.

First off, if you have a budget, I would opt to get R Studio Pro - it can handle the distribution of compute jobs between least busy nodes.   However that is not parallel computing, but rather job management.  Parallelism is usually the job of the libraries you build your compute jobs on.  So this becomes rapidly a software AND hardware architecture AND economic conversation.

How many cores / RAM do you think you would need in parallel?  In terms of the atomic pieces of work, what do they looks like?    You can purchase off the shelf servers with up to 96 cores ( the HP 500 series can have a quad CPU configuration with 24 cores.   You can also get them with 1 TB of RAM and many TB of SDD or hundreds of TB of rotational storage.  That is a lot of compute, storage, and the rest,

You also need to consider how you back this up, especially if you have large data sets or intermediate workprodyct you need to protect - if you spent 3 months of compute time and the data disappears, that's a bad day at the office.

If you need larger than this, then you are having to go with a cluster or going with infiniband or similar and creating a super computer.   This gets pricey. 

If you have multiple unrelated compute jobs or multiple researchers, where none needs a lot of cores in parallel, a cluster of linux nodes running R and using R Studio Pro to distribute the work may be an option.

If you are, OTOH, running a single job or where there are a lot of interlocked intermediate steps where you cannoy easily cut it up into parallel chunks, then a single linux instance as big as you can afford is the way to go. 

Your constraint to do this in house, not attached to the Internet means you exclude Amazon and their delicious C4.8xLarge ( 32 core / 60GB RAM ) instances.   If you could see a way to do it in the cloud, that is a very economical way to get compute when you need it.

So a lot depends on what "large" means to you.   There are certain computational methods which can easily lend themselves to a scatter-gather type workflows which an array of independant kernels can efficiently process a work to do queue of jobs more or less like an assembly line.  Other work lends itself to "one big honking computer"

Not sure if that is helpful.

-- Bob

I think it really depends on how much computing power you would like to have or you need. Since you mentioned "not connected to the outside world" and  provided you have a lot of computers in house , and would like to use them as local (cloud) grid,  you can configure for Hadoop/Map Reduce or MPI or other type of distributed computing (whichever you need) and use it.  

I think I need a lot more detail. Apologies for the "random thoughts" covering the various bases.

First off, if you have a budget, I would opt to get R Studio Pro - it can handle the distribution of compute jobs between least busy nodes.   However that is not parallel computing, but rather job management.  Parallelism is usually the job of the libraries you build your compute jobs on.  So this becomes rapidly a software AND hardware architecture AND economic conversation.

How many cores / RAM do you think you would need in parallel?  In terms of the atomic pieces of work, what do they looks like?    You can purchase off the shelf servers with up to 96 cores ( the HP 500 series can have a quad CPU configuration with 24 cores.   You can also get them with 1 TB of RAM and many TB of SDD or hundreds of TB of rotational storage.  That is a lot of compute, storage, and the rest,

You also need to consider how you back this up, especially if you have large data sets or intermediate workprodyct you need to protect - if you spent 3 months of compute time and the data disappears, that's a bad day at the office.

If you need larger than this, then you are having to go with a cluster or going with infiniband or similar and creating a super computer.   This gets pricey. 

If you have multiple unrelated compute jobs or multiple researchers, where none needs a lot of cores in parallel, a cluster of linux nodes running R and using R Studio Pro to distribute the work may be an option.

If you are, OTOH, running a single job or where there are a lot of interlocked intermediate steps where you cannoy easily cut it up into parallel chunks, then a single linux instance as big as you can afford is the way to go. 

Your constraint to do this in house, not attached to the Internet means you exclude Amazon and their delicious C4.8xLarge ( 32 core / 60GB RAM ) instances.   If you could see a way to do it in the cloud, that is a very economical way to get compute when you need it.

So a lot depends on what "large" means to you.   There are certain computational methods which can easily lend themselves to a scatter-gather type workflows which an array of independant kernels can efficiently process a work to do queue of jobs more or less like an assembly line.  Other work lends itself to "one big honking computer"

Not sure if that is helpful.

-- Bob

First of all, thank you for taking your time to give answers. What I have is a main server running probably scientific linux, 8 core cpu, several terrabytes of raid storage, and a flock of pc with giga ethernet, 32GB ram, 8 core cpu with no storage. All nodes will boot from main cpu, and as processing power need increase I will add more nodes to the flock. Main program running will be R, and python on main cpu. It is not the cutting edge hardware, but that is what we can efford for now. It is a basic diskless linux cluster. The question on my mind is, should I go for a cloud cluster, does it give me any advantage or existing setup will be good enough for now?

T&T