Data Mining of Big Data using tools like SVM, clusters, trees, MCMC, and NN have emerged in place of conventional statistics in order to handle large size and complexity. These are well-suited to *spot patterns in large, static data sets*.
But, the inevitable demand for *rapid analysis of streaming data* reveals the limitations of Data Mining methods, especially where data streams are unstable, chaotic, non-stationary or have concept drift. And that covers many important areas (!), like human behavior (economic, social, commercial, etc.) My focus is mostly computational finance.
Data Mining methods lag in adjusting to changes in the observed behavior. A key problem is the uncertainty whether estimates calculated from past data still are good enough to apply in the often changing future. How frequently and when does the model for prediction or classification need to be updated? How responsive to incoming data should the estimating procedure be to achieve the needed reliability, without getting whipsawed or lagging amidst shifts?
Having a machine learning tool that self-corrects to minimize prediction and classification errors is the challenge. A forgetting factor, as in the dynaTree R package, could be effective if it adjusted automatically. A gain factor, as in Kalman Filtering, can be set pretty well for steady systems (physics), but is sluggish in chaotic settings. GARCH and its relatives provide particularly clumsy structures. Many other approaches exist like Dynamic Model Averaging, adaptive ensembles. Some models must work well, like real-time demand estimators within Google’s Borg, which load-balances its servers.
Have you had success in this area? Can you cite methods, sources, examples or software? I would be glad to discuss this more if you have interest. Thanks.
Tom
Tom, in machine learning terminology your problem is referred to as “concept drift” or “drifting concepts”, which means that the window size has to adapt in time because the forecasting error changes over time in unforeseen ways. More information on that and appropriate methods are described for example here:
Tsymbal, A. 2004. The problem of concept drift: definitions and related work, Technical Report TCD-CS-2004-15, Department of Computer Science, Trinity College Dublin, Ireland, April 2004 (available at http://www.cs.tcd.ie/publications/tech-reports/reports.04/TCD-CS-2004-15.pdf).
Scholz, Martin and Klinkenberg, Ralf. Boosting Classifiers for Drifting Concepts. In Intelligent Data Analysis (IDA), Special Issue on Knowledge Discovery from Data Streams, Vol. 11, No. 1, pages 3--28, 2007.
Appropriate software for drifting concept modelling is RapidMiner supplemented with its extensions, e.g. Instance selection-, Feature selection-, Weka- and R- extensions.
Cheers
Pekka
Pekka, yes, concept drift is also well addressed in Zliobaite, I., 2010; Learning under Concept Drift: an Overview; http://arxiv.org/pdf/1010.4784v1.pdf
Among R-extensions, I favor dynaTree with its full array of online learning controls and analyses. Have others out there found packages that work in real-time applications?
The problem I find - in practice - is that such promising machine learning approaches do not produce sufficiently accurate, reliable predictions. When applied to unstable data, like market behavior, their predictions go astray when their parameters (like the forgetting factor) get mismatched to the changed, unseen data-generating process.
The challenge I see is to identify the approach that accommodates such changes or self-adjusts. As an real-life example, I would imagine Google’s Borg (server load-balancer) must be able to estimate real-time whether a spike in demand warrants re-allocating servers or filtering it as noise. Do others of you have suggestions?
Tom
For forecasting complex processes in a real uncertainty recommend looking systemically-event approach
http://inex-ft.com.ua/mds.php?name=Projects&pa=showpage&p_pid=52
Prompted by your publications and the above link, I will explore the potential for applying fuzzy technology to the problem. Thank you, Victor.
A good real-time classification method is presented in this thesis:
"Trend or No Trend: A Novel Nonparametric Method for Classifying Time Series".
http://web.mit.edu/snikolov/Public/trend.pdf
I implement Snikolov's method, and I don't think it can predict stream in data since we have local spike and global spike as well. Maybe I am wrong.
Regarding your question I think it is important to notice, that structural information which covers the relation between several data sources from which streaming data is obtained should be considered in predictive model. I work in the combination of time series and and network analysis methods, implemented for Apache Hadoop to enable both, large scale and also (near) real time analysis. We started with Hadoop.TS and switched to Apache Spark recently. In this context I think it is important to switch from a tool-centric to a data-centric perspective. Metadata management is another crucial factor. This drives our development of Etosha, a cluster spanning metastore which wraps a layer of contextual data about the technical metadata managed in Hive or even Kite.
I would like to discuss this aspects more, especially in the practical context of real applications, but also for further research.
Cheers,
Mirko
- Badges
- Science topic
- Similar topics
- Engineering
- Electrical Engineering
- Signal Processing