I would like to work in optimization of algorithms to classification, clustering and logistic regression for data mining over big data.
Cool Merry, I am a Data Mining researcher and am looking for algorithms to study and improve, can say to me what method do you use? This operation takes how much time?
Not sure what the scope is here, but here is something that may help.
We use big-data frameworks to run data-mining methods everyday - but just want to let you know that the industry practice today for the combination is not a "synergy" - it is more of a "connector".
What this means is:
1. Right now there exists no distinct set of algorithms specially designed for big-data (at-least not something that can capture the essence of big-data and take advantage of its CAP properties)
2. The killer is "integration" between big-data and data-mining than the individual algorithms' performances.
3. As Merry pointed out correctly, creating the algorithms and their implementation is not at all reduced in anyway by the big-data facilitation. They still take as much time as before (8 months in Merry's case)
4. The big-data realization, specially in the context of data-mining, is a more hardware demanding task. Not something that can readily be tested in traditional lab/academic setting. Amazon, Rackspace etc. are few helpers here - but a really cost viable scalable decentralized open cluster is still a distant dream for big-data researchers. For example, the feasibility of testing your new recommendation-engine's responsiveness for 1 million concurrent users in a lab setting, with enough detailing logging for its precision and recall
Improving any of these can help the industry/research.
- GK
http://gopalakrishna.palem.in/
Please take a look at Vowpal Wabbit ( https://github.com/JohnLangford/vowpal_wabbit/wiki )
I believe that array processing is a crucial part of data mining algorithms. So, check SciQL at http://dl.acm.org/citation.cfm?doid=2463676.2463684.
It can bridge the gap of data storage. In big data, you probably can't store everything in the memory. But storing the data in the file could be really painful in production.
I have some experience working with BIG data; see examples below:
Jiang B. (2015, accepted), Wholeness as a hierarchical graph to capture the nature of space, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1502.03554
Jiang B., Yin J. and Liu Q. (2014, accepted), Zipf’s Law for all the natural cities around the world, International Journal of Geographical Information Science, xx(x), xx-xx, Preprint: http://arxiv.org/abs/1402.2965
Jiang B. and Miao Y. (2014, accepted), The evolution of natural cities from the perspective of location-based social media, The Professional Geographer, xx(xx), xx-xx, DOI: 10.1080/00330124.2014.968886, Preprint: http://arxiv.org/abs/1401.6756
Jiang B. and Ma D. (2015), Defining least community as a homogeneous group in complex networks, Physica A: Statistical Mechanics and its Applications, 428, 154-160.
Jiang B. (2015b), Geospatial analysis requires a different way of thinking: The problem of spatial heterogeneity, GeoJournal, 80(1), 1-13.
Jiang B. (2015a), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.
- Badges
- Science topic