I used a classification method on a data set which was having 2000 columns and 62 rows.I understand this method can be applied on larger data too.

Cool Merry, I am a Data Mining researcher and am looking for algorithms to study and improve, can say to me what method do you use? This operation takes how much time?

1. I used Apriori-Hybrid for the classification of microarray colon cancer dataset.

2. I worked with one of the clustering method called K-modes for Luekmia microarray dataset also.

Each took me nearly 8months of time for implementation.

Great ! Tks Merry

Not sure what the scope is here, but here is something that may help.

We use big-data frameworks to run data-mining methods everyday - but just want to let you know that the industry practice today for the combination is not a "synergy" - it is more of a "connector".

What this means is:

1. Right now there exists no distinct set of algorithms specially designed for big-data (at-least not something that can capture the essence of big-data and take advantage of its CAP properties)

2. The killer is "integration" between big-data and data-mining than the individual algorithms' performances.

3. As Merry pointed out correctly, creating the algorithms and their implementation is not at all reduced in anyway by the big-data facilitation. They still take as much time as before (8 months in Merry's case)

4. The big-data realization, specially in the context of data-mining, is a more hardware demanding task. Not something that can readily be tested in traditional lab/academic setting. Amazon, Rackspace etc. are few helpers here - but a really cost viable scalable decentralized open cluster is still a distant dream for big-data researchers. For example, the feasibility of testing your new recommendation-engine's responsiveness for 1 million concurrent users in a lab setting, with enough detailing logging for its precision and recall

Improving any of these can help the industry/research.

- GK

http://gopalakrishna.palem.in/

Please take a look at Vowpal Wabbit ( https://github.com/JohnLangford/vowpal_wabbit/wiki )

Thanks a lot Gopalakrishna and Hadi

I believe that array processing is a crucial part of data mining algorithms. So, check SciQL at http://dl.acm.org/citation.cfm?doid=2463676.2463684.

It can bridge the gap of data storage. In big data, you probably can't store everything in the memory. But storing the data in the file could be really painful in production.

I have some experience working with BIG data; see examples below:

Jiang B. (2015, accepted), Wholeness as a hierarchical graph to capture the nature of space, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1502.03554

Jiang B., Yin J. and Liu Q. (2014, accepted), Zipf’s Law for all the natural cities around the world, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1402.2965

Jiang B. and Miao Y. (2014, accepted), The evolution of natural cities from the perspective of location-based social media, The Professional Geographer, xx(xx), xx-xx, DOI: 10.1080/00330124.2014.968886,  Preprint: http://arxiv.org/abs/1401.6756

Jiang B. and Ma D. (2015), Defining least community as a homogeneous group in complex networks, Physica A: Statistical Mechanics and its Applications, 428, 154-160.

Jiang B. (2015b), Geospatial analysis requires a different way of thinking: The problem of spatial heterogeneity, GeoJournal, 80(1), 1-13.

Jiang B. (2015a), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.