Your question is not adequately posed: 10 jobs of what size, doing what? at the very least, you need to capture some data from realistic runs of your target applications - you need to know memory footprints, get some idea of their transient storage requirements, etc.

if a job uses a modest number of cores, you should definitely get one big, beefy server, since that will make setup and maintenance much easier. if "job" means >=100 cores, then you're looking at a smallish cluster (32-100 nodes), and this opens up a variety of additional concerns: interconnect, storage, management.

the standard HPC interconnect is, without question, Infiniband. this is somewhat unfortunate, since IB is gratuitously different from ethernet, and although not exactly hard to accommodate, certainly clutters the landscape with additional concerns. most sites connect each node to one or more ethernet networks (usually one for IPMI, and often one for administration and/or booting). the IB fabric is often not bridged to ethernet, though it can be. skipping IB (because your jobs fit on a single node) is plausible, but probably unwise: not only do you want a fast network for storage, but it's substantially less efficient to schedule a random set of jobs onto only whole nodes. (occasionally running MPI over gigabit is usually asking for trouble, since ethernet latency, even 10Gb, is very, very poor compared to the compute power of a node...)

you almost certainly should provide a large, fast globally-shared storage resource. Lustre is, if not the standard, quite commonly used, and on the largest clusters. whether you provide node-local storage at all is up to you: it's largely an effort/cost tradeoff. disks are so cheap that it seems a shame not to order nodes with them - but that does not imply that you install onto them: applications may find local disks useful for checkpoints. don't be tempted to rely on existing storage (departmental servers, old-fashioned SAN, random NAS, etc) - they don't mix well with HPC.

for any cluster more than a handful of nodes, I strongly recommend booting nodes over PXE/NFS-root. that is, nodes are essentially stateless, and most admin tasks a constant-time operation. (some people prefer stateful nodes and then rely on a web of configuration-automation tools - this would make a lot more sense if PXE/NFSroot was somehow difficult, expensive or inefficient.) obviously, you need a scheduler unless your user and usage patterns are extremely rigid. all the schedulers are basically equivalent, and none of them are very good.

HPC centers normally avoid gold-plated "enterprise" hardware - culturally, HPC tends to prefer to spend money on the compute side rather than SAN, SAS disks, blades, etc. IB is better than any SAN, and instead of SAS, you get 5-10x more capacity via SATA, and blades are the same hardware in more expensive enclosures. these are largely aesthetic choices, and you may not have any freedom there.

ballpark: $10K/node including IB. any non-absurd storage system should manage to cost less than 4x the street price of bare SATA (currently $40-80/TB). software licenses, of course, can absorb any amount of money you let them. obviously a facility needs someone who can grok the whole thing (though one person can easily support many hundreds of nodes, and dozens of applications).

unless you're very careful, you should be using rack-mounted servers in a dedicated space with adequate cooling (a generic 1U compute node dissipates about 250W these days, so 40 of them in a rack is 10KW. old-fashioned datacenters will choke on that kind of power/heat density. it's easy to manage - the electrical feeds and cooling just need to be thought about.

Talking from my biological data analysis background, and my experience of building 3 computer systems for research use: I start by estimating following factors:

Disk space:

1. How much data on job can generate.

2. How much of that I need to store.

3. How often the jobs are run --> Data growth rate.

Processing power/Processors, Disk speed

4. How long I want for lifetime/wall-time of a running job

5. How many concurrent jobs/users the system will serve.

Data connection and networking

6. Does the system have to transfer data around and how?

RAM

7. Balance with all above factors are I will decide how much RAM I need.

Software specific requirments

8. What are acceptable performance requirements for the software I run.

Other things one can consider is how the HPC system will work from users' points of view. Will they work with a grid, a cluster or workstations?

These things help me to figure out MINIMUM requirement for the system. For actually system, I would add up 30% to 50% of the total power of the system so it has room for expansion. In addition, if jobs and data growth rate is high, I have to make choice whether to buy plenty now or later.

Then I will contact to Dell, HP or PenguinComputing or home HPC vendors to ask for quotes. With Dell, you can get a good idea of budget buy going to their homepage and build up components yourself, and usually they have substantial discount for academic. Contacting the hardware vendors, talk about your needs and ask for quotes is a very good way to build up your system. They may also have similar customers that they can share some information.

Also my experience is that, asking around online is one good step, but not enough. I found it is practically useful to go "corporate style": make field visits, ask for presentation/consultant from vendors.

Hope this help.

E5 series processor, 32 GB RAM, 6TB HDD, Fast Bus Speed, 10 cores/ threads processor if not interested in E5 series

2 years ago we tried to build something simillar, but did not get the funding. Ansys with some extra modules (2 licenses optimizer, 1 or 2 HFSS, ) for 10 machines would cost us around USD 22k-25k, academic licenses

@Vang Le. Thanx. These are very useful tips. I am going corporate style already, but want to get a feel from academia.

Ok, here is another question. Matlab only supports 12 cores. Would it make sense to have more cores per client for later expansion?

I would suggest you to subscribe the beowulf mailing list and search its archives (they are public). It's one of the best resources available for these kind of questions.

Regards

Are you also interested to electromagnetic design (antenna modeling, antenna siting, electromagnetic compatibility, radiation hazard etc) tools?

@Ricardo, thanx.

@Mauro, yes.

From your question, it doesn't appear that you will be running many in-house applications. Is that true? As such, you will be looking at an x86 solution based on your list of supported applications. Vang Le did a terrific job of summarizing what you should be looking at.

In-house applications may benefit from CPU architectures other than x86, depending on how well they scale and how much time to run.

Well, there will be inhouse applications based on our inhouse Matlab code, if that is what you meant. I don't know why you think an x86 would suffice.

Your question is not adequately posed: 10 jobs of what size, doing what? at the very least, you need to capture some data from realistic runs of your target applications - you need to know memory footprints, get some idea of their transient storage requirements, etc.

if a job uses a modest number of cores, you should definitely get one big, beefy server, since that will make setup and maintenance much easier. if "job" means >=100 cores, then you're looking at a smallish cluster (32-100 nodes), and this opens up a variety of additional concerns: interconnect, storage, management.

the standard HPC interconnect is, without question, Infiniband. this is somewhat unfortunate, since IB is gratuitously different from ethernet, and although not exactly hard to accommodate, certainly clutters the landscape with additional concerns. most sites connect each node to one or more ethernet networks (usually one for IPMI, and often one for administration and/or booting). the IB fabric is often not bridged to ethernet, though it can be. skipping IB (because your jobs fit on a single node) is plausible, but probably unwise: not only do you want a fast network for storage, but it's substantially less efficient to schedule a random set of jobs onto only whole nodes. (occasionally running MPI over gigabit is usually asking for trouble, since ethernet latency, even 10Gb, is very, very poor compared to the compute power of a node...)

you almost certainly should provide a large, fast globally-shared storage resource. Lustre is, if not the standard, quite commonly used, and on the largest clusters. whether you provide node-local storage at all is up to you: it's largely an effort/cost tradeoff. disks are so cheap that it seems a shame not to order nodes with them - but that does not imply that you install onto them: applications may find local disks useful for checkpoints. don't be tempted to rely on existing storage (departmental servers, old-fashioned SAN, random NAS, etc) - they don't mix well with HPC.

for any cluster more than a handful of nodes, I strongly recommend booting nodes over PXE/NFS-root. that is, nodes are essentially stateless, and most admin tasks a constant-time operation. (some people prefer stateful nodes and then rely on a web of configuration-automation tools - this would make a lot more sense if PXE/NFSroot was somehow difficult, expensive or inefficient.) obviously, you need a scheduler unless your user and usage patterns are extremely rigid. all the schedulers are basically equivalent, and none of them are very good.

HPC centers normally avoid gold-plated "enterprise" hardware - culturally, HPC tends to prefer to spend money on the compute side rather than SAN, SAS disks, blades, etc. IB is better than any SAN, and instead of SAS, you get 5-10x more capacity via SATA, and blades are the same hardware in more expensive enclosures. these are largely aesthetic choices, and you may not have any freedom there.

ballpark: $10K/node including IB. any non-absurd storage system should manage to cost less than 4x the street price of bare SATA (currently $40-80/TB). software licenses, of course, can absorb any amount of money you let them. obviously a facility needs someone who can grok the whole thing (though one person can easily support many hundreds of nodes, and dozens of applications).

unless you're very careful, you should be using rack-mounted servers in a dedicated space with adequate cooling (a generic 1U compute node dissipates about 250W these days, so 40 of them in a rack is 10KW. old-fashioned datacenters will choke on that kind of power/heat density. it's easy to manage - the electrical feeds and cooling just need to be thought about.

Mark Hahn's answer is definitely educating for people with some experience in HPC, who are aiming for hardcore HPC systems. People in fact has different ideas about what an HPC system is. For a starter like Bilal Siddiqui, he probably ends up buying a system that can fit in a rack, not in a 30m2 room, or 300m2 floor.

I have a system where I do heavy computation - De novo assembly, sequence alignment - on one SMP server and do lighter jobs in others - single-threaded command line, viewing result. I use a SAS disk array attached to the main server with 24Gbit SAS connection, others connect to it with NFS. This system offers a versatile environment where I can do tiny Grid computation, resource-intensive GUI applications, Desktop environment, etc.

For computation intensive application, I tend to favor GPU-utilizing applications over cluster/grid-utilizing applications. This gives me a powerful rack-size HPC system.

But please don't take me wrong. I did not imply that rack-size HPC systems do not need cooling and you can put next to you in the room. Cooling, UPS, and backup are critical to keep your hardware and data safe. How to setup and maintain either storage or backup devices deserve a whole topic itself.

With today's dense manycore systems, you can pack a lot of computational horsepower into a single node containing multiple CPUs. You can also add an accelerator, such as a Xeon Phi co-processor, or a GPU, or both. The Phi is sometimes called a 'cluster on a chip' and Intel has 3 different Phi product lines:

http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html

I've recently purchased a system with 164 GB of ECC RAM, 16 regular CPU cores (2 x 8-core Xeon CPUs), a Phi with 57 cores, and an Nvidia K4000 GPU for another 768 CUDA cores, all for about $10K, in a single box. I'm not sure if MatLab can be configured to take advantage of a Phi or a GPU yet, but if not, they should be supporting these accelerators in the near future, given the direction the hardware is moving.

If you don't want to go the accelerator route, you can just buy a node with four of the higher-end CPUs. Intel has 6, 8, and 10 core CPUs these days, so put four of 10-core CPUs on the same motherboard with lots of memory and you can have a 40-core system in a single box.

The single box eliminates the need for an interconnect, network storage system, etc. It also lets you use normal wall-socket power and normal room climate control. You'll still want UPS and backup systems, though these can be much smaller scale than for a cluster.

The upshot is that you may not need a networked cluster any more -- thanks to Moore's Law, you can pack a lot of computational horsepower into a single box today.

@Joel Adams: Thanks for mentioning Phi coproessor. I came across it before but it did not catch my attention. Reading from the link you gave, I have following clarification questions if I may:

1. Is this Phi the 'GPU' rendition of Intel? I feel there the purpose is the same, but there maybe lot of differences in architecture.

2. About programming languages we have: OpenCL for ATI GPUs and NVIDIA GPUs; CUDA for NVIDIA's, and what is it for Phi?

3. Is it true that any pararallel application (multithreaded, multicore, or multiprocessor) that can run on Xeon chips will run with Phi? If this is true, it is a very attractive feature. And how much of tuning has to be done before recompiling for Phi?

Intel calls the Phi a coprocessor rather than a GPU. The "shader" cores on a GPU have historically been a bit limited in what they can do -- they don't handle branching (if statements) very well, their double-precision floating point performance was pretty limited, until the most recent version of CUDA, they did not handle recursion at all.

My understanding is that the cores in a Phi are full Xeon CPU cores, without the limitations of GPU cores, so that algorithms that run on normal CPU cores will run exactly the same (or better) on Phi cores. Moreover, each Phi core supports 4 hardware threads, so a 61 core Phi can actually handle 244 software threads at the same time. I believe each of these cores also has a vector unit to parallelize vector computations.

In terms of software, the Phi supports MPI so multiprocessing applications written in MPI for a cluster should run out of the box on a Phi. OpenMP is also supported, so multithreading applications written for shared-memory systems using OpenMP should also run unaltered. OpenCL is also supported so OpenCL applications should also work. I think the same goes for OpenACC.

The box I ordered is supposed to arrive this week; I'm anxiously awaiting it so I can try all of this out! 8^)

@Bilal - my comment about x86 implies 64bit - as all Intel/AMD processors are.

Your list of commercial applications (including Matlab) heavily favors that Intel architecture (as opposed to Intel's Itanium, IBM's Power, Oracle's Sparc, etc). GPU exploitation also tends to be Intel oriented. Purchasing a machine from Cray or SGI still gets you an Intel solution - albeit different than from Dell, HP, or IBM.

The point I was trying to make was for you to recognize that your application requirements might restrict your system/CPU choice.

@Jerry, Ah, this makes much more sense now. I have little knowledge of computer architecture due to my mechanical/aerospace background. Thanx for the info.

@Joel, but don't u know adding all power on a single motherboard removes redundancy in the sense that once that system is out of order the entire lab goes down with it.

@Joel Adams, thanx for the suggestions. Yes, Matlab supports GPU:

http://www.mathworks.com/discovery/matlab-gpu.html

@Vang Le, thanx. Your inputs have been insightful, as usual.

@Bilal - you asked about the fact that Matlab only supports 12 cores.

The question to ask is how many Matlab users will you be supporting, and how many will be running simultaneously. A single user, or a series of users who will only run Matlab in a "one at a time" queue won't require as many cores as if multiple users want to run Matlab simultaneously.

When it comes to determining what kind of HPC machine/environment to purchase, you need to be looking at all your requirements. There are Intel chips with 12 cores per socket, but the more common are 8 and 16 cores. More is better (in aggregate), because even if you're not using everything now, the user community will grow their demand until such time as they exceed whatever hardware you've put in place. Buying to satisfy today's needs is unfortunately going to only get you 12-18 months (at best) satisfaction from your user community. As commercial software is updated, support for more and more resources (cores/memory/disk) will only be increasing.