We have a research project in which we will generate about 1TB of stream of data per day.
So for this I wonder if we can use distributed framework to process the data.. if it can be done so apart from the economic reasons, what would be the advantages.
You could use Condor HTC: http://research.cs.wisc.edu/htcondor/. I have used this for data-intensive computations. It is very easy to install and maintain a local cluster of available workstations.
Maybe Condor is fine, but Globus is another option, and you can run Condor (condor-G) with Globus.
As Yehia said I think a bespoke solution will work out best here, though you can rely on many established tools. You can use common fast file/data transfer tools (e.g. Aspera [which is commercial] or a well-tuned GridFTP) to effectively move the data from source to the processing hosts, and then use a different toolkit to take care of the processing. Depending on the characteristics and compute requirements of the processing codes, as well as the expected rate of change in the design your processing codes, you could opt for Condor HTC, the QosCosGrid computing environment, or a Python-based framework using, for example, Fabric.
- Badges
- Science topic