I want to process semi structured data inside a Cloud platform. the preprocessing scripts are developed with JAVA , Python and Bach languages. the final step is to store the output data into a Hadoop cluster. what is the best solution to combine Cloud and Hadoop in the same time ?
you mentioned your approach to do.. i understand but i want clarification of questions need to be asked one step back i.e. why do you want to do this? and what is the use case?
You Can use any of the below cloud platforms which enables you to connect with Hadoop as well.
1. Google Cloud Platform
2. AWS
3. IBM Blu-mix
and there are still more....
If you combine Cloud and Hadoop primitives then performance will suffer as both these systems have their own software architecture that is meant to run independent of each other. There are other ways of doing what you mentioned in your question.
(1) Transfer files (this is going to be slow)
(2) The other way round is given here: https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-overview
You may be able to use some commands to transfer data back to your Hadoop cluster.
(3) There will be security issues with your case as well, as outlined here:
https://stackoverflow.com/questions/34207593/downloading-files-from-google-cloud-storage-straight-into-hdfs-and-hive-tables
thank you for your answer. i think the third method is more efficient. i will try to read and apply it.
Hadoop could also do the work. If you insist using Cloud to preprocess the data, then use some data ingestion tools of Hadoop: Sqoop, Storm, NiFi.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python