The amount of data that is processed annually has exceeded the zettabyte boundary. This is a number that would only be mentioned in highly theoretical articles 2, 3 decades ago. Such insurmountable amount of data gave birth to a new term : big data. What do you think is the most important tool that will allow us to handle this explosion of data?
a) Do you think, it is the increase in the performance of CPUs, which is currently significantly slower than the growth of data?
b) Do you think, it is the new programming languages being introduced, which will make processing of this data much easier?
c) Do you think it is some novel data analytics algorithms that will shed light into significantly easing the handling of data?
d) Or, anything else ?
e) Or, are we dead in the water and there is no hope ?
These are practical approaches to deal the BigData.
(1) MapReduce (HADOOP),
(2) BSP (HAMA)
(3) STORM (Real time processing)
In my opinion, your problem belongs to the area of Combinatorial Optimization. If it is indeed, then we deal with the Complexity Theory of algorithms as classes P and NP. Thus, I'd chosen the point c) from your five points. I think, that only effective polynomial-time algorithms (from P-class) can be used in practice in order to solve BIG DATA problem.
These are practical approaches to deal the BigData.
(1) MapReduce (HADOOP),
(2) BSP (HAMA)
(3) STORM (Real time processing)
I would say that it would be advances in parallelization whether in the form of different nodes, use GPU's or multi processor machines . As we have bigger amounts of data our current sequential algorithms will only slow things down. This in turn will have an impact on programming languages.
So, if I am hearing this correctly, 1) programming models like MapReduce to handle BigData ..., however, since they will always be very compute-intensive, 2) massively parallel platforms to process the algorithms in (1), such as GPU-based or, multi-core, multi-socket rack-based datacenters ...
In summary, BOTH hardware and software are necessary.
I think real time distributed/parallel processing (Hadoop) will do and only results can be stored processed data can be discarded. It can be like stream processing.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data