Here are few pointers, how we help our customers deal with this problem:

1. Bigdata, as many define, has Volume, Velocity and Variety attributes. Except for volume, which requires massive infrastructures, research on Velocity and Variety problems need no more than tradition lab settings (at least during the research phase)

2. Big-data in itself is useless, if not "applied" somewhere - which is the 4th V, people typically refer to, known as, 'value add'. These 'application problems' are, most of the times, just regular problems that just got 'scaled up/out'. So, research on these problems, once again, need no more than traditional lab settings.

3. When both of the above scenarios fail for your particular research case, the world anyway has "free for one year" small instances of AWS and Azure, that should satisfy any reasonably well defined big-data problem.

GK

http://gk.palem.in/

Thanks Gopalakrishna for your interest and valuable answer.

Could you pls explain the Velocity and Variety attributes in more detail? Or could you pls recommend several papers on it? I'd like to see if it's really possible for me to research on these two attributes of big data.

In your 2nd point paragraph, you said "... that just got 'scaled up/out'" , so what's the meaning of 'scaled up/out' ?

Looking forward to your further reply on the above issues. Thanks again!

You are welcome Bin Liu.

Velocity and Variety, as the names imply, refer to the 'speed' of the data arrival/processing and 'nature/type' of the data being processed.

For example, a sensor generating data from a fast moving vehicle (notifying the speed of the vehicle every second), or traffic arriving on google search, are good examples for the 'velocity' of data. The biggest challenge in such cases is 'how to achieve low latency and high throughput'. Even a single slow component in the processing pipeline can bring down the whole system. This is where stream-processing can help.

Multiple-sensors generating different 'types' of data from the same fast moving vehicle (notifying the engine temperature, audio feed from tail-vibration, video feed from any attached camera on the front) is an example of 'variety' of data. The biggest challenge in this case is dealing with all the different types (audio, video, key-value pairs, time-series etc.) in a unified way. This is where NOSQL can help.

Social meda data is a very good example of both 'velocity' and 'variety', since the data arrives at a very high speed (thousands of tweets / sec) and with high variety (text, link, image, video...). Processing social media data for deriving the 'value-add', the 4th V, is the example of 'scaled up/out' problem.

To understand it, consider a simple tweet of text and doing NLP (text mining) over it. Its a pretty simple straightforward CPU-bound process. Now, instead of one tweet, imagine 1 million tweets to process. The fundamental problem (digesting the tweet with NLP) has not become any complex (in terms of algorithmic complexity) - rather it just got bigger. So, instead of using one computer, if you use 1 million computers in parallel, you should be able to solve it as 'easily' as the initial problem. (If you really want to understand it, contrast it with solving a 2-color problem vs 3-color problem in graph coloring - where just scaling out and spawning new computers will not make the problem any simpler).

In other-words, these big-data 'value-add' problems have always been there for quite a number of decades (e.g. estimating the no. of people who are going to signup for your discount deal) - but their 'scale' got bigger in the recent years (that is, instead of dealing with 1000 customers, now they deal with 1 million customers) - which is perfectly solvable, just by adding more computers to work in parallel. (Which is not the case with certain fundamental algorithmic problems, where adding more computers, would not reduce the complexity of problem in any significant manner).

So, to do research on these kind of 'value-add' problems, one would not really need 100 computers, instead you can just start with 1 computer, solve a small 'instance' of the problem and put it production on 1000 machines to run in parallel, and the chances are you are already solving the 'bigger' instance of the same problem. (Well, there are certain restrictions that make this invalid, but in most of the cases where big-data is being applied today in the industry, this stands valid)

The recent implementations we did for our customers in big-data on production scale, utilizing all of these 'velocity', 'variety' and 'value-add' attributes may give some more insight into the nature of these problems, please check them at: http://gopalakrishna.palem.in/Portfolio.html#portfolio

Thanks Gopalakrishna for your so detailed an explanation.

Your work on big data is pretty cool and impressive for me .

I am reading this question and related reply very late. But still want to extend the question asked by Bin Liu that

" Large IT companies like Google, facebook, Microsoft, Baidu all have their facilities and massive data supporting commensurate research on big data analysis".

How individual researchers can get the "Big Data" and work on it? Please suggest.

Thank you for the kind words, Bin Liu.

Madhavi, If the problem is 'obtaining data', then there are different techniques (ranging from traditional web-scraping to creating pseudo data by one-self) that vary with domain.

Especially in 'natural language processing (NLP)' (and all its peers like sentiment analysis, trend analysis, intent analysis etc.) - social media data is very popular these days as the data-source. There are both commercial data aggregators such as 'Radian 6' (some of our projects use this), as well as free web-scraping tools such as Apache Nutch (basically a crawler, but can be customized to retrieve only portion of what you want).

Few links to publicly accessible data-sets are provided at: http://my-classes.com/2014/05/27/interested-in-big-data-research-but-not-sure-where-to-get-the-data/

GK (Gopalakrishna)

http://gk.palem.in/

Thank you so much Gopalkrishna. Will check this link and get back to you for more suggestions.

There are tons of free data available on the Internet such as Brightkite checkin data, and nighttime images; see examples below:

Ma D., Sandberg M., and Jiang B. (2005, accepted), Characterizing the heterogeneity of the OpenStreetMap data and community, ISPRS International Journal of Geo-Information, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1503.06091

Jiang B., Yin J. and Liu Q. (2014, accepted), Zipf’s Law for all the natural cities around the world, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1402.2965

Jiang B. and Miao Y. (2014, accepted), The evolution of natural cities from the perspective of location-based social media, The Professional Geographer, xx(xx), xx-xx, DOI: 10.1080/00330124.2014.968886,  Preprint: http://arxiv.org/abs/1401.6756

Jiang B. and Ma D. (2015), Defining least community as a homogeneous group in complex networks, Physica A: Statistical Mechanics and its Applications, 428, 154-160.

Jiang B. (2015a), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.

Thanks Prof. Jiang. I'll check the information you provided.