Suppose I have a big project and I want to distribute modules to different partners and then recollect it again, how can I do it?
Similarly as what BOINC is doing. I don't want to use any tool or software. I would like my own development, but don't know how to get it done.
If you plan to use a only a programming language : Java and serialization and Java middleware such as JMS/Java RMI can be suitable.
If you plan to develop eveything in C or C++ you need to implement some essentiem concepts such as:
- Message Oriented Middleware to deal with asynchronous interactions
- A kind of virtual machine to be able to distribute your own modules without problems of execution on heterogeneous platform.
You can use a broker/subscriber model with an AMQP broker like RabbitMQ. You write a client which subscribes to the broker and looks for tasks/input data to compute, then submits the results to a web service when it completes. Celery (a python project) provides some of the heavy-lifting to interface with various broker backends.
Doing such a thing by yourself it is challenging, for sure, and could bring you a lot of insight in parallel and distributed processing . Nevertheless, there is a lot of work done in this area that deserves to be reused, even if this is about software libraries or tools. Perhaps looking from different perspectives to your problem could help you better define your question:
- grid middleware (Globus Toolkit 5.x for example) - for placing a sequential or parallel task/module/job on some remote resource and for retrieving its result; the task would be technology/programming language independent;
- metaschedulers (e.g. GridWay)
- distributed environments and programming models (e.g. Message Passing Interface) - esp. for C/C++ and/or Fortran codes;
- distributed systems (in the sense the previous replies indicate, i.e. messaging systems);
- an so on.
A better scoping of your question could bring you more focused answers.
I am using Erlang/OTP to develop distributed applications. A lot concerning distribution becomes trivially easy (i.e. getting clearance to communicate across fire walls is most of what needs to be done). And it is fun to program in Erlang. Look at www.erlang.org and learnyousomeerlang.com for starters. You also get hot code updating, high availability and the most stable virtual machine in the domain. Picking the right tool is always a good idea.
Thanks you all,
Actually I started working as I posted this ques on 6th.
I have considered Map reduce concept to implement it in Go language for this..
Another approach you could take into account is to use mobile agent technology: it makes easy to distribute tasks, is robust, and you would have a uniform view of the processing.
According to Carlos, mobile agent is a good paradigm for project. It can be easily implemented in Java using serialization and remote method invocation. Otherwise some mobile agent middleware can be used (e.g., Agilla).
from the distributed System perspective, you can use Remote Procedure Techniques such as RMI, CORBA.
Also you can use web service models and from that you offer many services such as replication
- Badges
- Science method