Most of the stuff we hear today is, unfortunately, by people who do not understand the nature of the historically unprecedented transitional period we are living in. And I don't mean any of the notorious "singularities". I mean the scientific transition to a fundamentally new, structural, form of data representation.

In particular, in this context, the situation with more data *that cannot be adequately represented*---and hence adequately processed---is much closer to your comparison to "a tsunami".

My view is shaped by the realization of the inadequacy of the conventional, numeric, forms of data representation, which are of *spatial* origin (going go back to the ancient measurement practices) and hence are inadequate for the 'intelligent' information processing.

To be able to fully benefit from the enormous amount of data, we need to rely on the new kind, structural, data representation.

I could not agree more. Most probably (hopefully) both, statistics and data visualization will gain more attention than ever.

Coming from the geospatial information science, I am completely with you considering the need for innovative and specialized visualization techniques, which will help to analyze, interpret and understand the huge data steams coming in. Abstraction and generalization with certainly have their roles in this.

Without having bright ideas on that topic, I also keep wondering if the required processing could be done by computers alone. It might be at the time to take a step back in automation and re-consider the possible role of humans in the loop. In the end a human mind is not too bad in pattern recognition etc. - assuming that the available data is represented in a well digestible form, which might vary between individuals.

Without too much self promotion, might I point you to another question 'How important is storytelling and visualization for your science': https://www.researchgate.net/post/How_important_is_storytelling_and_visualization_for_your_science. I would very much like to read about your view on this!

The promise of the early part of this century of genomics and microarrays is similar. Traditional approaches to statistical inference for microarrays, for example, meant a lot of false relationships were "discovered" due to the lag between technology and methodological development. Probably cannot be controlled but there is always rubbish generated with new advances until we see what actually works. One of the fears is that Big Data will overtake careful experimentation in the same way that the chase to sequence the genome overtook the careful approach of the past century. Whose to say which one will lead to more truth?

All of the above! :-) These are not mutually exclusive. Big Data is both real and a hype. Whether the extra attention it now gets will be as productive for society and business as claimed, nobody can say for sure, but I'd argue it looks pretty good. At least the results that are now emerging in my own research area (databases) look already promising and I see no reason to believe this will stop. Will it cause a paradigm shift? Very doubtful in my opinion.

Fair enough. In fact, I am quite exited about the directions this will take in the area of 'Observation Web', i.e. the connection between physical sensors, scientific models/simulations, and user contributed content. Data coming from all of the before mentioned technologies grow quite amazingly over the last years and heterogeneity keeps raising.

To my understanding the main challenge will be to identify synergies and increase the collaboration with other research fields - such as the data base, statistics or data visualization community - in order to avoid re-invent the wheel. If we continue working on own solutions, we will very soon run out of resources and loose focus.

I totally agree with Sven in the sense that Big Data problems will also require more collaborative efforts to solve in a timely manner. The hope is that more rapid turnaround on promising manuscripts and more online journals with more open access will aid in reducing the number of parallel efforts. Competition for the monies that will allow for these efforts, however, is a major hurdle. The current process of grantsmanship and monies that would be released to solve these issues are of major concern. The grants have to be written to remove the "siloes" to level the playing field so more good ideas can be investigated. Unfortunately, this isn't currently the way things work with the NSF and NIH.

Valid points. Open access and maybe also what is called science 2.0 will certainly influence the scene. I would expect that current roles (and therewith responsibilities and duties) will undergo changes towards cross-disciplinary works, citizen will get more involved in data and feedback provision, and those holding classical silos (in data or models/simulations) will have to reveal their resources. Others will begin to integrate information from multiple sources in order to create value added services - where added value might be in business terms, social or environmental benefits. One of the larges questions might be: how will the big institutions be involved in this game and how will they change? ... but this is another story.

Several research initiatives in Europe move into this direction, for example the Future Internet - Public Private Partnership (FI-PPP) initiative and also parts of the next major funding framework, Horizon 2020 that is. Still, also many of those typically applying to the related calls are a bit too conventionally thinking to take big steps and some risk coming with that... however - slowly or rapidly - some exiting thinks will certainly emerge from all of this.

"How will the big institutions be involved in this game and how will they change?" ... Great question. Teaching is already taking a big turn with great courses now being offered in the Open Classroom setting. Research dollars fromt he NSF and NIH can't continue to support Research Universities at the level they now do. The gov't is moving rapidly already towards more industry involvement with Beltway companies getting grants that used to go to the Universities. Data security is a definite roadblock. Definitely changing rapidly.

Absolutely. Much of this can only happen bottom up, although with funding from major players. The European Commission for example (under the above mentioned Horizon 2020) established a flagship initiative that is called 'Youth on the Move'. It more and more seems that we need a cultural shift across large parts of the society. This partially happens anyway due to the use of technology in young generations, but still might be directed. Topics, such as Big Data and the required capabilities (incl. statistics, visualization and others) should be part of that.

Also relating to the increasing involvement of industry in research, and again in Europe, we witness quite a series of complementary study modules focusing on innovation in diverse domains. For example, three seven year initiatives (on energy, climate change and ICT) help universities setting up related study programs... exiting times.

From a scientific standpoint it is hype, but at the same time it also highlights some of the constraints that the scientific community need to address. For the most part, machine learning algorithms, and visualization techniques do not scale well as the data gets bigger. Most publications deal only with sampled data and do not address big amounts of data processing which in itself is a big problem.

Good point. We now get much data at our fingertips - being a threatening tsunami, or the new oil to fuel innovation...

This raises the importance to clearly define the context and boundaries of any specific research and articulate why a particular data set suites a study and others don't. While this was always an aspect of experiments, it now becomes one of the priorities.

I do not see an issue when a publication only deals with a small sample of a data set, but it now becomes more important to take this as a basis for (i) scaling the experiment to the full set (or a larger part); and (ii) connecting the sample to data from other sources. In this sense, also these two dimensions - which already existed previously - provide new challenges and opportunities.

Little follow-up: a new article in the guardian closely related to some of our discussions: http://www.guardian.co.uk/news/datablog/2013/apr/25/forget-big-data-small-data-revolution.

Here is another article in popular media covering related matter: http://www.wired.com/insights/2013/04/big-data-fast-data-smart-data/

I will present some related (more technical, concrete) thoughts in my talk: "Transforming Big Data into Smart Data: Deriving Value via harnessing Volume, Variety and Velocity using semantics and Semantic Web" Abstract gives some concrete pointers: http://www.sebd2013.unirc.it/invitedSpeakers.html

Talk is about ready and will post it in 2-3 weeks on slideshare, if anyone is interested.

Opportunity

Absolutely Big data makes the results more reliable. In this way, the results wouldn't be affected by special situation. It could be generalized.

Big Data is the buzzword of the year. No more, no less.

It is an opportunity if you are looking for a job, or a funding.

If it is not the case, you can work as usual on Very Large DataBase, data mining, distributed or parallel processing, stream mining, online classification...,

but you will say to the taxi driver that you work on Big Data, or more trendy, on FastData.

I agree on this one, Raphaël.

There is a trend to have more and more (heterogeneous) data available from diverse sources and it is not bad to name this 'phenomenon'. In the end, this will (or better should) not alter our daily research work. Still, apart from possible disruptions, we might in some cases benefit from newly released information - if we find it ;-)

By the way, I gave up chasing for a single model (and format) that suites all. In the end, data is collected and optimized for a specific purpose. If you intend using it for something else, you have to life with some needs of conversion and be careful about its fit for another purpose than it was initially collected for.

Seems to me that with the ability to arbitarily combine data from multiple heterogeneous sources, preventing Type I error is going to be murder. Perhaps it's better to treat searching for correlations among heterogeneous sources as ways to generate testable hypotheses?

Sven,

I think that an important benefit of "Big Data" is:

Using massive parallelism framework such as Hadoop, it is possible to deploy models which are based on detailled data rather on aggregated data.

The consequence is that we can add to the modelization process, an automatic features construction and selection process.

We have tried this approach on the first version on the Orange Customer Analysis Platform (see my page to find the paper), and it works.

To me.. always we have encountered huge amount of data to be processed, however, it seems that companies that provide computational services (development) always need to re-invent the wheel (like..consultant companies), specially in computer science. For example, Knowledge Discovery in Databases (KDD) in early 90s', later Data Mining, then Business Intelligence (2000 aprox), then Business Analytics (2008), 2009-2010...Big Data.... (flavors of these: Web Text Mining, Web Content Mining, Web Structure Mining, Web Intelligence, Social Web Mining, Social Intelligence, Customer Intelligence, Process Mining, etc.). All these concepts were THE HOT WORD of the year... and all of them means the same.

The idea of the area in KDD, was to show that companies have "huge amounts of data" that was not used. Therefore, we can apply several techniques to gain knowledge to take better business decisions. Of course, with AT386 and AT486 processors.. or servers not much better than that... "huge data" was restricted to the data that we could process in a decent time... (i.e. before a decision needed to be taken)... of course, online social nets, even the web didn't exist at that time, so we processed commercial data, production data, etc.

Several years passed... in fact...almost 2 decades passed... now every one talks about twitter, facebook, of course you can do text mining to gather interests from people taking.. (that are all my papers about, since 2004).. but once again.. we have "huge amounts of data"... this time hardware allow to process it (using map reduce...or whatever other ´new' technology like HANA to load all in RAM...(before that was too expensive...today you can go for it....))

Just to close my comment... in my opinion, this is neither an opportunity, nor a burden or a trap... this is just the natural evolution of computer science, that companies put a "new name" to try to sell (again) a solution they probably have developed in the past with an old hardware.

The true opportunity goes to regular people that will have to work with these systems to truly find patterns, learn behaviors, discover preferences, etc. You must learn the skills and topics that enable you to do so...

Cheers