Pertaining to the need of data integration (or higher as data governance), a hybrid solution of data warehouse and big data is proposed in:
http://assets.teradata.com/resourceCenter/downloads/WhitePapers/EB-6448.pdf
I think that this solution is really interesting and hope that this solution can be a big trend in the future. I hope that some of you have experencies for such solution and can share it (eg. data integration centers and implemented technologies, basic theories, required standards).
Thank you!
Data Warehouse (DW, or EDW or FDW) is an architectural concept in data computing. Bigdata Analytics? To me, it is just a Data Analytics. Often, Bigdata is used to scare customers to sell a product, especially in healthcare. Let’s talk about this one by one. Data Warehouse: Industry consider Bill Inmon, Ralph Kimball are the pioneers. Bill Inmon Defines: A data warehouse is a subject-oriented,integrated, time-variant and non-volatile collection of data in support of management's decision-making process. For example Sales, Marketing, HR, Claims are Subject Areas of an organization. Integrated means: EDW will embrace Data integration tools (Informatica, Sync Sort) to extract the multiple data sources in organization and structure them based on Subject Area and other logical business entities. Time-Variant means Historical data is kept in a data warehouse. Once the data is in the EDW, it will not change (Non-Volatile) Ralph Kimball defines: A data warehouse is a copy of transaction data specifically structured for query and analysis. EDW stores the data in a normalized or denormalized (star or snow-flake schema). This is highly time-consuming and expensive. Sometimes I refer this as structuring the data Who is right? They both are: Organizations adopt either or a combination depending upon the use case(s), Data Quality (De-Duplicate, Pseudonymization, Matching, and Merging), Master Data Management, Reference Data Management, and Data Governance are the capabilities or data normalization techniques will be built on top of Data Warehouse with technology as enablers to those capabilities. Business intelligence processes, tools, and technology will be used on top EDW to provide reports, dashboards, ad-hoc analysis and predictive analysis. This is the Analytics portions of the data. Now talk about Bigdata Analytics: This is much-hyped word in the industry in a sense that it became a marketing jargon for both vendors and career seekers. In the modern, innovative business era, variety of sources ingesting at a high volume and high speed (millions of rows in seconds). Traditional tools and techniques adopted in the EDW world are not efficient. Modern tools like Hadoop, Machine learning tools running on commodity hardware is helping integrating the data into EDW with cost benefits. These modern tools’ efficiencies helped the engineers to come up with schema-less analysis, and changing business models are pushing towards late-binding with a less early-binding. Binding is integrating business rules into the data. Most of the time analyzing the data using these modern tools on top of huge sets is referred as BigData analytics.
Thanks for reply!
So I'm very interested in the term of "modern tools" as mentioned in your reply. I found the similar approach of "modern DW" defined in:
(1) https://www.cio.com/article/3176947/data-warehousing/recharge-your-knowledge-of-the-modern-data-warehouse.html
(2) https://www.cloudera.com/products/data-warehouse.html
One of difficulties is that this approach is not well documented and standardized?!! How do you think about this opinion? Do you have some projects or documents related to this approach?
Thank you!
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data