What is essential is understanding the nature of the data. Nothing beats Exploratory Data Analysis. See John Tukey's Exploratory Data Analysis. Addison-Wesley Publishing Co., 1977.

Try and assimilate (not only knowing, assimilate means making it a mother tongue) the ideas of multidimensional statistics: principal component analysis, cluster analysis, multidimensinla scaling. You should be able to translate any problem in terms of rows(statistical units)/columns(variables) and then describe the correlation structure you find in both row ans column space in terms typical of the problem at hand, something like 'the variables mostly loaded on the first principal component are X1 nad X2 so PC1 can be interpreted as..'.

It is thirty years I do this job and it worked well in fields ranging from physics to psychology, it is like working with a 'general script' that can be adapted to different situations but you need humility and fllexibility and more important of all..keep it simple...(attached some more technical considerations on this theme).

What is important to keep in mind is that in this approach the crucial point is to collect a good (balanced not necesarily big) data set, if you have a good data set with a correct blend of the 'order parameters' (or hidden dimensions, concepts, tendencies call them as you want) you can obtain a clear picture, otherwise you will do a disaster, the work is exactly the opposite that the classical work in physics: you will have the laws emerging from the data and not the other way around, in the attached paper we derived in this way the concept of complexity.

I'm sorry, the file is attached here...

What is essential is understanding the nature of the data. Nothing beats Exploratory Data Analysis. See John Tukey's Exploratory Data Analysis. Addison-Wesley Publishing Co., 1977.

Thanks for the quick replies. Very good to see that Big Data as a buzzword does not jeopardize well established disciplines. It is an important point to make, even 'Big Data' appears in the titles of hundreds of events this year, underlying principles remain!

Still, getting more concrete now, which would be the software tools to use for Big Data management, analysis and visualization? Are there any front-runners?

SAS is in my opinio perfect for 'hard core' statistical computations at the basis of Big Data but the problem is that is not so easy to use as for graphics modules, usually I make the computations with SAS the I export data files to SigmaPlot for graphics, but in my field I have not so crucial needs of graphical data representations ...

Just wanted to supplement AG's excellent reply with the following:

If you have Big Data, it is likely you do not only have bi-linear data f(x,y) but you would most likely have multi-linear data f(x,y,z,...). Please know that the PARAFAC model is mathematically unique while also being (highly) parsimoneous and multilinear. This means that PARAFAC does not suffer from the rotational ambiguity in contrast to the bi- and multi-way PCA analyses.

HOWEVER: simple color-coding plotting of the raw data will often reveal the major trends in data - but correct visualization becomes complex with highly complex data - in particular if (some of the data) are categorical and/or the axes are non-obligue.

I strongly recommend reading Peter Huber’s reflections on data analysis: “Data Analysis: What Can Be Learned from the Past 50 Years.” This famous statistician, one of the fathers of the robust statistics, critically explores several crucial issues related to Big Data Sets and classical statistics or data mining. By the way, robust procedures for analysing Big Data Sets – seem to be a very exciting challenge!

R is as comprehensive as any statistical package and it's free. It takes a little time to learn but it's worth it. R is particularly strong with graphics.

I recommend reading the book "Doing Data Science" from O'reilly.

The tools and software you'll find handy are Apache pig for you data processing, cleaning and basic analysis.

R and Python for you more advanced analysis and modelling.

I use Tableau for my visualization and find it quit good. However as it has been always said R is a great choice for visualization as well.

Thanks for the recommendation Aida. In fact, I bought this book earlier this year and started reading ;-) Certainly worth it!

I think what Alessandro said above about having a balanced set is important. There are the questions of time (how long a section of time or a time T analysis slice? ), space (forms or shapes of components of the set and there relations (do they overlap, etc.) but there is also the CULTURAL ANALYSIS of BIG DATA so well described a month ago by Bentley, O'Brien & Brock, here in "Mapping collective behavior in the big-data era", in Behavioral and Brain Sciences (BBS) vol.37, Cambridge press.

I am not sure if I can give you people link here this way but here it goes :

https://www.researchgate.net/publication/261235670_The_Global_Shift._The_Shadows_of_Identifiability_Behav_Brain_Sci._%28cf._pp._99-100_109_111%29_Invited_peer_community_commentary_on_target_article_Bentley_O%27Brien__Brock_Mapping_collective_behavior_in_the_big-data_era_Behavioral_and_Brain_Sciences_%28BBS%29_vol.37_Feb._63119_Published_online_by_Cambridge_University_Press_26_February_2014

if not, it is the first one featured in full-text on my page.

Colin T. SCHMIDT

Assoc. Prof. Appropriateness of Technology

LeMans U. & ParisTech

https://www.researchgate.net/profile/Colin_Schmidt?ev=prf_highl

Article "The Global Shift. The Shadows of Identifiability", Behav Br...