I would like to set up a computational laboratory. I would like to know how much it would cost (including electricity, maintenance) to start up a computational laboratory with 200 processors to begin with.
Ahmed Farid has done a good job of identifying the main decisions you need to make. What you decide affects the answer to your question. ARM processors generally consume less electricity and generate less heat than x86 processors, so which you choose will affect your need for power and cooling.
One other power-related issue is how reliable your local power grid is. If it is not very reliable (e.g., power failures or brownouts), you'll want to include a battery-based uninterruptible power supply (UPS) into your design and use it to power your cluster, or you'll lose all your work in progress any time there is a momentary power failure.
If you want to build it from inexpensive system-on-a-board components (e.g., Raspberry Pi), I would look at Nvidia's Jetson board. It's more expensive than the Pi, but has a quad-core CPU and a 192 CUDA core GPU, so it delivers quite a bit more processing power. There are other options out there too.
The networking decision is an important one that depends on the nature of the programs you'll be running. If the programs running on your system will be doing lots of communication between the distributed processes, you want a high-speed low-latency network, or those processes will be doing a lot of waiting for messages to arrive.
One other question you need to resolve: do you need 200 processors (CPUs) or 200 cores? With 10 or 12 core Xeon CPUs, you can build a 200 core system in a relatively small form factor (e.g., half a rack). The challenge becomes providing enough memory and network bandwidth to keep them all busy.
And once you have designed your cluster, built it, and installed it in a room with sufficient power and cooling, it is very helpful to have a system administrator to maintain it. For a small cluster, this can consume 1/4 to 1/2 of a full time equivalent (FTE) position, depending on how many people are using it, requiring software installation, etc.
I agree with Ahmed and Joel. However, when you decide what you want to buy, look for the good supplier. In a kind of the tender specifications you should put a condition of self-turn off when the cooling system fails. Looking for such a supplier it is good to ask some people from your place who have bought similar computer system. Following this way, I bought a computer system with 320 cores few years ago, I have got the good system administrator and no problems with its operation.
I think that a good way to start on a computational laboratory of hundreds of cores, the use of GPU system or GPGPU is the best way to learn parallel and distributed computing. the cost of the GPU card of 384 GPUs is about $500, the programming language is CUDA (parallel C++). This technology is growing to gain its place in the high performance computing domain, if this is your preoccupation. good luck.
Yes, decide whether you want to go the Cuda route - almost certainly the answer is no, since Cuda will only offer speedups to quite niche domains, and requires substantial programming effort to use. (The cost of a 384 "core" Cuda-capable card is about $100. However, these cores bear almost no relation to real processor cores - not commensurate.)
For conventional processors, you face a fundamental decision: cheap and flakey versus rock-solid but somewhat expensive. In other words, a pile of desktop-quality machines, or the more "enterprise-y" server approach.
Desktop machines will normally occupy more space, with braindead designs, especially for cooling. Desktop machines normally integrate a low-quality 1Gb NIC, and usually have significantly less efficient PSUs. Desktop machines will usually have just one CPU socket, often limited memory capacity, and won't support ECC. Desktop hardware, including disks, will rarely survive its warranty period, which is 1-3 years.
Servers will be rack-mount devices with sanely-engineered airflow, ECC and remote control (IPMI). Nowadays, servers have 94% efficient PSUs, which are usually even rightsized (unlike desktops, where PSUs are normally >2x oversized.) Server hardware will, unless abused, easily provide 24x7 service for > 5 years.
Besides aesthetics, I think part of the decision is based on scale. If you meant 200 cores, then the desktop approach is still mostly viable, and can save substantial capital cost. Something like 25 8-core machines is not too big, and although it will certainly consume more manual labor, it won't keep a box-monkey constantly running. 200 processors (sockets) via desktops *will* be a challenge to maintain.
One of the best things of the server approach is that it's HIGHLY scalable. A single technical person can easily handle all maintenance for thousands of server nodes, since the basic failure rates will be substantially lower and IPMI provides powerful managability. (I'm assuming Linux here, of course - doing clusters with anything else is begging for pain and misery and expense.) Servers will also allow much nicer hardware, letting you configure cores, memory, disk, interconnects to your heart's content. 200 cores is almost not worth doing via the server approach, though (~8 servers), but 200 sockets will be 1-2 racks.
Size your cooling for about 150W/socket. Typically desktop systems will go into deeper power-saving modes than servers when idle (maybe 20W/socket), but you have to design for peak power consumption/dissipation. Remember that office-type cooling is normally only good for a few hundred watts (any sane datacenter handles 15 KW/rack without any heroic efforts at all.)
Strategically, I would start by buying a rock-solid server-grade box that you use mainly for RAIDed storage and management. You don't want your primary storage farmed out to desktop-level NAS boxes (or, horrors, USB-connected disks!). Then some sort of interconnect - at least gigabit, but do consider at least some 10G ports. Then add compute nodes, either of the desktop or server variety. For a lab of more than a handful of hosts, you'll definitely want to standardize and automate: I'm a huge fan of NFS-root servers because they are infinitely easy to manage (but some people like stateful nodes and script-based config/management ala Chef/Puppet/etc.) Ideally a lab should have a user/login node kept separate from the admin/storage node (probably also separate from compute/job nodes.) Personnel requirements depend primarily on how complicated your environment is: whether you want lots of different software stacks, frequently-updated, multiple user-groups, etc. Managing 200 servers and a modest single-domain software stack for one workgroup should only take a small fraction of one competent FTE.
1. The cluster will be utilised for running Molecular Dynamics simulations ( such as GROMACS or GROMOS) & Quantum chemistry softwares (for electronic structure calculations). We are aiming to run the simulations for nano - micro seconds level. Also, we want to develop softwares our own & testing of these will also be done. Other researchers using the cluster want to run computational fluid dynamics simulations.
2.We would like to have Linux/Unix system to perform simulations & developing codes.
So far I haven't observed power shortage, but local researchers say during winter the power might be interrupted. This means a strong back up would be needed.
* I will continue after studying good replies here so far.
ARM is simply an architecture traditionally associated with the phone/tablet world - it is therefore approaching from the low-power (and low-speed) direction. Although x86_64 and ARM have different histories, there is no significant, technical difference in their capabilities or power-efficiency. Currently, ARM is trying to break out of the mobile market into the "webscale" market - basically servers for things like webservices, memcached, nosql, where computational performance is not the metric of concern.
Although you can get some early ARM server-class hardware, it is not terribly attractive from a price or performance standpoint. It may compete better in the future, though it's hard to imagine why the x86_64 world would not also improve.
x86_64 is definitely the way to go if you do not already have some specialist knowlege or interest in ARM (or GPUs).
Your most recent message mentions power protection. This is a good idea, but can be quite expensive. Most competent and mature codes (like GROMACS) are designed to write checkpoints and can be restarted efficiently. Academic clusters often do *not* provide backup power for compute nodes, for this reason. The best choice depends on the properties of your power, though, since simple UPS-based power protection (which will add something like 20% to your system costs) will only support operation for a handful of minutes. To protect against longer outages requires much more expensive infrastructure (room-scale UPS with automatic generator). Of course, your server/storage infrastructure should definitely have UPS support, since it should comprise only a small fraction of your power dissipation (relative to compute nodes).