Can we combine cluster computing and cloud computing? That is, in the cloud environment if we want to balance load of VMs on physical machines, can we apply concepts of cluster computing to the cloud environment? What software is available to accomplish this?
Depends what you are trying to accomplish. For simply large scale data processing tasks take a look at Hadoop. But if you are doing clustered parallel processing in more time sensitive applications then Storm Cluster is great. Both can run out-of-the-box on AWS. If its simply a matter of load balancing, you can do this natively on AWS to manage load across VMs connected to an Elastic Load Balancer.
Depends what you are trying to accomplish. For simply large scale data processing tasks take a look at Hadoop. But if you are doing clustered parallel processing in more time sensitive applications then Storm Cluster is great. Both can run out-of-the-box on AWS. If its simply a matter of load balancing, you can do this natively on AWS to manage load across VMs connected to an Elastic Load Balancer.
Cloud computing can be seen as an umbrella term that encompasses many other concepts, including clustering computing. Therefore, you can have, for instance, a cloud made of multiple clusters or a cluster and a normal server. Examples of software: Opennebula, openstack and Eucalyptus (all of them allow you to add or make internal clusters for your cloud).
About the load of VMs, it depends on the virtualization solution you are using and the cluster you have. If the hypervisor see the cluster as a single server (again, is the type of cluster you use) you can run a VM sharing the load (automatically made) otherwise, for the softwares I know, is not possible.
A cloud version of Galaxy is availble that creates cloud clusters or biomedical work. See http://wiki.galaxyproject.org/CloudMan
I love Cycle Computing's capability. They have a really nice layer added to typical cluster management products that deals with the cloud's unreliability. With Cycle you can set up a raw cluster to do any cluster workload. However, if your workload requires low latency interconnect, then you should not be using general purpose clouds like AWS, but instead you should use IB-based clouds, such as Sabalcore or Nimbus.net.
Are you confusing the term "virtualized cluster" with the phrase "cloud computing" ? Cloud computing has no direct relation with virtualized clusters or clusterig, but virtualization is connected to both the cloud and the virtualized cluster. You can use VMware ESXi 5.1 virtual cluster. For vapps, you my not need a virtualized cluster as they can be deployed directly on a standalone host.
BTW What do you intend to do by combining Cloud Computing with Clusterig ?
What exactly do you want to do? In principle, IaaS-provided virtual machines are indistinguishable from "bare iron" so you can use whatever cluster software you wanted to use with "real" machines.
For me the answer is simple: you should be able to provision, start, monitor, and tear down a virtual cluster from your laptop. The folks that do Cloudbiolinux and CloudMan do exactly that, and I think that should be the model for all computational science.
Here is the front page for that capability:
https://biocloudcentral.herokuapp.com/launch
What Cycle adds on top of that is the ability to manage a virtual cluster across many different geographies, and let AWS deal with the CDN and data replication aspects of large data sets.
The piece that is still missing, particularly with AWS, is the ability to implement hybrid clouds, so that you can keep a workload in-house if you have the capacity. Remember that cloud computing is about 3x more expensive per hour as compared to having your own gear. a $24k/year AWS large instance is a $10k server if you buy it for yourself.
I am working on OpenStack hybrid clouds for that exact use case, as are a couple dozen folks at the National Labs.
From the cluster computing perspective, MIT's StarCluster at star.mit.edu/cluster, allows automated configuration, deployment and management of a virtual cluster in Amazon EC2. StarCluster offers a simple command line interface that launches a master and multiple worker instances, installs Open Grid Scheduler (formerly Sun Grid Engine) as the scheduler and provides shared NFS filesystems backed by EBS Volumes. There are 32/64-bit AMIs that incorporate some standard cluster software and libraries e.g OpenMPI, Hadoop, Open Grid Scheduler, ATLAS, Scipy, Numpy, CUDA, MAGMA and PyCUDA.
For load-balancing, StarCluster's Elastic Load Balancer can be setup to dynamically add or remove worker nodes based on the number of waiting jobs in the cluster queue. The cluster may also use either EC2 on-demand or spot instances with a maximum bid price to dynamically scale the cluster according to cost.
Depends, what Cloud and what cluster you can connect, if you work with Microsoft Technology you must to use Microsoft System Center App Controller and Virtual Machine Manager.
You can provision cloud virtual machine (vm) instances and cluster multiple instances using MPICH to for a cluster under the umbrella of cloud computing. For a empirical comparison of cloud computing and traditional computing for cluster and other services, you can look at this paper: https://www.researchgate.net/publication/252628427_An_Empirical_Investigation_of_Cost-Resource_Optimization_for_running_Real-Life_Applications_in_Open_Source_Cloud
Conference Paper An Empirical Investigation of Cost-Resource Optimization for...
In our lab, we use ProActive middleware to perform distributed computing via a java paltform both on local resources (small clusters) and in the cloud (AWS).
http://www.activeeon.com/
http://proactive.activeeon.com/index.php
- Badges
- Science method