Hi!
What is the most effective way (in terms of development process, availability, costs) to create an API on Cloud which expose services that in background do Hadoop Map Reduce processing?
For example, let's say I want to build an API on Cloud and offer a number of services, count(String fileName) which opens a file and counts the number of word occurrences using MapReduce. The results of calling this service will be written to the storage, such as HDFS, Cloud Storage buckets, etc.
For example on Google Compute engine is possible to create a MapReduce appengine application (not API services exposed on Cloud) but is not very straightforward to create an endpoints application (backend API) based on that.
Any ideas are highly appreciated (do not limit to Google platform, any platform offering this would be ok).
Thanks
Sorin
Try using Hadoop hosted on Amazon Web Services. They have ready to use ecosystem and it is possible make API calls (REST based web services).
Hi Pankaj,
I do not want to call existing services provided by Amazon. I want to create my own services (API) that will be available in Cloud for other developers to call my API. Also I want that my services will use Hadoop MapReduce.
MapReduce/Haddop is a generic data parallel approach/platform, rather than applications. It is up to your imagination to decide what application funactionalities you want to provide as a service, such as your mentioned 'word account service' - using Haddop do something useful. In this sense, no one can help you, but yourself.
Regarding to the API design, it also depends on your service. However, since MapRead-related service is likely to process a large amount of data, I just remind of you that passing huge data as API argument seems not a good idea.
Good luck!
@Shiping: I was not thinking of passing Big Data as argument to functions calls but to pass name of file in HDFS where the data resides. Secondly, my question goal was to receive an answer from a developer view that provides a comparison of existing Cloud platforms (Google Compute Engine, Amazon AWS, Bluemix, etc) from the perspective of which is more efficient in terms of implementation of own API's that in background calls Hadoop services. Seems IBM's Bluemix is the closer to satisfy these requirements.
Please refer to:
Shiping Chen, Tomasz Bednarz, Piotr Szul, et al.: Galaxy + Hadoop: Toward a Collaborative and Scalable Image Processing Toolbox in Cloud. ICSOC Workshops 2013: 339-351
- Badges
- Science topic
- Similar topics
- Biological Science
- Physiology
- Physiological Processes
- Stress