I have read a couple of articles which are trying to sell the idea that the organization should basically choose between either implementing Hadoop (which is a powerful tool when it comes to unstructured and complex datasets) or implementing Data Warehouse (which is a powerful tool when it comes to structured datasets). But my question is, can´t they actually go along, since Big Data is about both structured and unstructured data?
It's very hard to answer this question in general without taking into considerations what your specific needs are. Also, "Data Warehouse" is a pretty general term which basically can mean any kind of technology where you put in your data for later analysis. It can be a classical SQL database, Hadoop (yes, Hadoop can be a Data Warehouse, too), or anything else. Hadoop is a general Map Reduce framework you can also use for a lot of different tasks, including Data Warehousing, but also many other things. You also have to bear in mind that Hadoop itself is a piece of infrastructure which will require a significant amount of coding on your part to do anything useful. You might want to look into projects like Pig or Hive which build on Hadoop and provide a higher level query language to actually do something with your data.
Ultimately you have to ask yourself what existing infrastructure is already in place, how much data you have, what the kind of questions are you want to extract from your data and so on, and then use something which fits your needs.
I am with the idea to implement hadoop because it can handle both structured and unstructured data. http://www.sand.com/hadoop-fits-big-data/
Hi Charles,
Actually Big Data could be seen as an evolution of Data Warehouse. Although both aim to analyze and extract valuable information from the data, Data Warehouses usually get their data from other databases. Big Data systems are more oriented to fetch their data from multiple sources.
Therefore, the kind of data sources can be a good criteria in order to decide whether to implement DW or BD. Surely both type of systems could be implemented and used together.
I strongly recommend you to read the following white paper from Oracle:
http://www.oracle.com/technetwork/database/bi-datawarehousing/data-warehousing-wp-12c-1896097.pdf
It's very hard to answer this question in general without taking into considerations what your specific needs are. Also, "Data Warehouse" is a pretty general term which basically can mean any kind of technology where you put in your data for later analysis. It can be a classical SQL database, Hadoop (yes, Hadoop can be a Data Warehouse, too), or anything else. Hadoop is a general Map Reduce framework you can also use for a lot of different tasks, including Data Warehousing, but also many other things. You also have to bear in mind that Hadoop itself is a piece of infrastructure which will require a significant amount of coding on your part to do anything useful. You might want to look into projects like Pig or Hive which build on Hadoop and provide a higher level query language to actually do something with your data.
Ultimately you have to ask yourself what existing infrastructure is already in place, how much data you have, what the kind of questions are you want to extract from your data and so on, and then use something which fits your needs.
Big Data is a term used to data sets whose size of data is beyond the ability of usually applied tools to capture, arrange and process the data within a tolerable elapsed time. But Data-warehouse is a collection of data marts representing historical data from different operations in the company.
For me, when considering Big Data, we have to think about how to process it. Hadoop consists of HDFS and MapReduce, so that it can not only store data but also have a paradigm to process it. Of course, there are several applications built on top of Hadoop like Pig and Hive. Warehouse is just a collection of database as others said before. It could have several data operations. But, this function is limited. For the data structured or unstructured, it imposes limitation on how you can process data. For warehouse, since it can handle structured data very well, it is a good choice. But for those unstructured data, apparently warehouse is not capable to handle it.
Certainly one can engineer a way to use both Hadoop and a data warehousing solution in one application. But I agree with comments that Hadoop is part of an evolution that goes back to data warehousing. Data wharehousing technologies were organized to "slice and dice" which meant to group data according to some subset of it's dimensions (attributes) then aggregate using the other dimensions. Hadoop generalizes the grouping concept with the map function, the aggregation concept with the reduce function, and, I believe, does a better job of distributing the map and reduce processing across large numbers of machines. That's how I think about the evolution.
Hive is the data warehouse built on Hadoop that have integrated many existing internal techniques of relational data warehouses along with its own query optimizations based on the MapReduce execution engine. Indexing should be the next major aspect to catch up with.
Check the latest publication http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-14-2.pdf
Meanwhile, modern Big Data processing requirements include not only data manipulation and queries, but also post-query mining and learning, as pointed out by the Spark group: http://www.cs.berkeley.edu/~matei/papers/2013/sigmod_shark.pdf
In my opinion, Hadoop (especially 2.0) provides a better integrated platform in this regard.
i think actually Hadoop is a very good platform to do some data-warehousing plus other data intensive works(YOUR LIFE would be very easier if you use Hive and Pig), so depending what you expect from Hadoop i suggest you can take them as equals.
The Hadoop ecosystem starts from the same aim of wanting to collect together as much interesting data as possible from different systems, but approaches it in a radically better way. With this approach, you dump all data of interest into a big data store (usually HDFS – Hadoop Distributed File System). This is often in cloud storage – cloud storage is good for the task, because it’s cheap and flexible, and because it puts the data close to cheap cloud computing power. You can still then do ETL and create a data warehouse using tools like Hive if you want, but more importantly you also still have all of the raw data available so you can also define new questions and do complex analyses over all of the raw historical data if you wish. The Hadoop toolset allows great flexibility and power of analysis, since it does big computation by splitting a task over large numbers of cheap commodity machines, letting you perform much more powerful, speculative, and rapid analyses than is possible in a traditional warehouse.
A Data Warehouse is a structured relational database, where you aim to collect together all the interesting data from multiple systems. When putting data into a warehouse, you need to clean and structure it at the point of insertion. This cleaning and structuring process is usually called ETL – Extract, Transform, and Load. The data warehouse approach is helpful because then your data looks clean and simple, but it is also very limiting: you’re now unable to change the questions you want to ask later since the data’s already been pre-processed, or to correct for errors in the ETL process. It can also get very expensive, as enterprise data warehouses are typically built on specialized infrastructure which becomes very pricey for large datasets.
Hello
Check it out this attachment. this is useful to u.
http://ir.inflibnet.ac.in/handle/1944/1890
Comparing Hadoop to a data warehouse is like comparing apples to oranges. The data warehouse is a concept and Hadoop is a technology. You can’t compare the two. As a concept of clean and integrated data, the data warehouse is here to stay. In the past, data warehouses have typically run on relational databases. However, over the last couple of years various limitations of the RDBMS have emerged (exploding license costs in the face of growing data volumes, poor fit for purpose for querying graphs and hierarchies and ingesting unstructured data types etc.). At the same time MPP SQL query engines on Hadoop have appeared such as Apache Drill that now make it possible to query data that sits on Hadoop. I would argue that there are clear signs now that the golden era of the RDBMS and data warehouse appliances is coming to an end. I have written a whole series of posts on the subject if you are interested in all of the details. Data Warehousing in the age of big data. The end of an era?
http://sonra.io/data-warehousing-in-the-age-of-big-data-the-end-of-an-era/
I imagine that theoretically, at least, Hadoop can do anything that you might do with a data wharehouse because the map part of map-reduce is so general. I'm not very knowledgable about data wharehouses actually, but it seems to me that the transform step of ETL is similar to map of map-reduce. But if map is more general (it can be any function) than the transform of ETL, then perhaps Hadoop provides more generality. And certainly being designed for a network with thousands of nodes also gives it more power.
Similar thing in my opinion. Just that Hadoop has the ability to process unstructured and large volume data.
Financially speaking, Hadoop can be a very inexpensive alternative to a data warehouse. You can store your structured data across cheap computing/storage nodes as opposed to adding large servers, disaster recover, fail-over, etc.
Where as in Hadoop, you can break up the data and let HDFS, the Hadoop distributed file system handle the 3X copies of each chunk of data. You can use Pig, or Hive, or Ambari or Flume to run queries against just as if you are using a DW. I have implemented this at a client site where we were migrating a COBOL/DB2 system and decided to go with Hadoop just to save money. It's also a great transition point until you know where the data should be housed, but Hadoop storing 3X the data and the ease of scaling out versus scaling up is a nice feature.
- Badges
- Science topic