Let me preface this by saying I have never used EC2, so my answers will be more general in context.

based on your description - max 100 nodes * 2days/per run * multiple runs a month, what would be the monthly cost to have a 100 node EC2 cluster? From your description that you may use the dedicated cluster for 6-10 (maybe more) days per month. Tearing it down and reinstalling the required software for each run may take upwards of a day on each end of your process, so that now puts you at 9-15 days (2 days for the run + 1 day for combined install/tear down).

Benefit of using EC2:

* you only pay for the computing you need

* no need for electrical power, maintenance, or local SysAdmin support

* available when *you* need it exclusively

Possible drawbacks

* you have to recreate it each time you want to use it

* data confidentiality issues(?)

* based on usage, eventually costs > purchase of similar hw

* network connectivity issues to the EC2 cloud

Things to consider:

* can you share the cost (with another researcher/dept) for the purchase of the 4 blades?

* you have to buy licenses to proprietary software regardless of EC2 or local

* network connectivity to campus backbone, or would the cluster be locally isolated

It really comes down to a cost-benefits analysis. I haven't used the amazon calculator, but it's not *just* the hw that you need to factor in for the calculations.

I'm sure you've thought of these things, but I wanted to put them out here just in case you hadn't - and to give anyone else a starting point for further identification of issues.

Let me preface this by saying I have never used EC2, so my answers will be more general in context.

based on your description - max 100 nodes * 2days/per run * multiple runs a month, what would be the monthly cost to have a 100 node EC2 cluster? From your description that you may use the dedicated cluster for 6-10 (maybe more) days per month. Tearing it down and reinstalling the required software for each run may take upwards of a day on each end of your process, so that now puts you at 9-15 days (2 days for the run + 1 day for combined install/tear down).

Benefit of using EC2:

* you only pay for the computing you need

* no need for electrical power, maintenance, or local SysAdmin support

* available when *you* need it exclusively

Possible drawbacks

* you have to recreate it each time you want to use it

* data confidentiality issues(?)

* based on usage, eventually costs > purchase of similar hw

* network connectivity issues to the EC2 cloud

Things to consider:

* can you share the cost (with another researcher/dept) for the purchase of the 4 blades?

* you have to buy licenses to proprietary software regardless of EC2 or local

* network connectivity to campus backbone, or would the cluster be locally isolated

It really comes down to a cost-benefits analysis. I haven't used the amazon calculator, but it's not *just* the hw that you need to factor in for the calculations.

I'm sure you've thought of these things, but I wanted to put them out here just in case you hadn't - and to give anyone else a starting point for further identification of issues.

Spot instances would not work for us, so we calculated using the \

on-demand price, and it was cheaper to run our own.

If the tasks are at least nearly independent, you really only

need a compute farm, and can forgo an expensive Infiniband switch.

Are you running 100 nodes, or 100 cores? If nodes, how much memory

and cores per node.

We have 4 SMP SuperMicro machines each with 64 cores (Opteron Interlagos

processors) and 256GB of RAM. They allow us to use lots of compute

cycles and run the occasional large shared memory problem.

You could also share time and costs with another group, to allow you to

get more equipment, since Torque would allow you to set fair usage policies.

That is essentially what the cloud is doing for you, sharing costs and compute time.

Cloud, or those infrastructure/platforms adopting virtualization may not be good for HPC, unless neither latency nor QoS is your concerns. As for large scale exploratory research with no customer experience guarantee requirement, I would recommend you to work with your own cloud, e.g., those built on top of OpenStack.

I don't see why you can't build a reusable image, then gen up as many t1.micro instances you need to run your calculations. They're FREE. It's unclear from your request as to how intense your computing needs (e.g., memory vs. CPU vs. storage), so if you're doing this on a shoestring, this method should cost little to none.

Thank you all for your very useful answers so far.

James: I think ~100 (physical) cores (computing independent tasks) would already be useful. I am sometimes still a bit puzzled how cores are counted (virtual or physical). In the SuperMicro machines, would you see 64 cores using torque? Sounds like a viable solution to me, then.

Kai: QoS is less of a concern since (at least my) processes do not need to share memory.

Shlomi, very interesting suggestion regarding t1.micro instances. I will discuss it with my colleague. I suppose the image includes a whole OS? That sounds though a little heavy in usage to me.

Yes, that includes an Ubuntu-based Amazon Linux distro. And you can get an _approximate_ core count here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html.

Cheers!

Thanks Shlomi - I'm sorry, but it's still not quite clear to me: do I for each simulation run (and each change of parameters!), have to upload an entire image containing OS (ubuntu) plus simulation program to each micro instance? That would seem very heavy on network traffic to me ... Or, do I upload the image with OS just once. Thanks for your help!

Create an image for a t1.micro that includes everything you need for it to run. Then create multiple instances of that image. There's tons of documentation on the AWS site to help you with that.

There are limits for how much free t1.micro instance time you can get (750 hours/month for the first 12 months). Erik, I'm curious if you've done anything with this in the past few months.

There are several aspects to this questions. First, I'm not certain but assuming you have taken a look at the MIT Starcluster tool for creating clusters on AWS?

http://star.mit.edu/cluster/

The star cluster template is essentially an AWS script that orchestrates the creation of the cluster. It uses SGE, but you could set up your own script to create the cluster of your liking. The AWS tool you will be using is CloudFormation, and that is essentially all StarCluster is, a CloudFormation config script.

Second, if you are running jobs that are primarily independent of each other, this is a good fit for cloud. In particular, I'm part of a project right now that has scaled up to about 8000 instances (not cores) for running this type of job. The instances run on spot, so you will have to devise a general strategy of how to take advantage of it. For example, the Virginia East region in the US is usually very busy for spot, but other regions like Oregon are not. This can give you better availability and pricing. Also, if you can diversify the types of instances you are able to run on, that will increase spot's usability for you.

http://star.mit.edu/cluster/