Mining the data in real-time on a data stream and being able to use that for short-term prediction is an interesting topic. What are possible tools you have used and can you suggest, which one is the better one? A scenario would be that data is coming through web services so would be wonderful if a tool can be integrated with the data as it comes on web services.
As far as I know there just two tools that are the most well-known in the DS community. These are: (1) VFML, which freely available here http://www.cs.washington.edu/dm/vfml/ and (2) MOA, which is freely available here http://moa.cms.waikato.ac.nz .
However, VFML is currently unmaintained (since 2003!) and MOA requires a little bit of expertise with Java/WEKA (don't worry, the manual is very clear).
I used both and I think MOA is the most powerful. As it is integrated in a Java framework, you can easily modify (rather extend) it to your purposes without too much messy coding.
Hope it helps!
Dear Mr Khan: there are multiple approaches to processing real-time data. Eg: Massive Online Analysis (MOA) http://moa.cms.waikato.ac.nz/ for RT data-streams, even you can make the process of knowledge engineering/discovery for a specific topic, build your knowledge-models and predict on-demand (enrich the knowledge base with new cases is your decision, maybe an Incremental Classifier). Thinking about your scenario, I suggest to you to take a look at data virtualization, Data As A Service (DaaS), and real-time/on-demand ETL if you are using a virtual datawarehouse or a swap area (database).
Best regards.
There is a survey article and an Excel table about data mining tools in general, but not focussed on data stream mining:
https://www.researchgate.net/publication/220080654_Data_mining_tools
https://www.researchgate.net/publication/256384390_Updated_list_with_data_mining_tools_%28Supplementary_material_to_the_survey_article_Data_Mining_Tools%29
It might be help to identify interesting candidates.
Article Data mining tools
Data Updated list with data mining tools (Supplementary material ...
If you are going to use ETL/SOA, then Rapid Miner can also help, but then you will need a full, rather than free, edition.
You could apply your data mining algorithm over the input data streams using a data stream management system (DSMS).
SciKit Learn (for Python) has some streaming/on-line algorithms.
Off the top of my head I can think of MinBatchKMeans for clustering and HashingVectorizer for feature extraction from text. Some of the linear models can be updated as new data comes in.
http://scikit-learn.org/stable/
Dear Mr Khan, perhaps take a look at "The Top 7 Big Data Streaming Analytics Vendors – Compared" (http://searchdatamanagement.bitpipe.com/data/document.do?res_id=1412267046_362&li=1047083) can give you a new perspective.
Best regards.
The greatest asset I derive from data mining is that I get to use my mind in so many different ways...sometimes mining huge data sets pushes me into areas of interest I never thought I would have...sometimes I find what I'm looking for - and at other times I find myself so surprised by what I do find...those are the fun times...you really never know what else you might find...
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining