I read somewhere earlier: after a final model is built with available data and used in "production", it is possible to add unseen, misclassified samples to the model to enhance the future predictions. Is there a name for this method?
In more details with steps:
1. Gather and explore your data, build features, estimate the model accuracy with cross-validation.
2. Build the final model based on the available samples.
3. Start to use the model in real environment and some new samples will be misclassified for sure since there is no perfect model.
4. These new, unseen, misclassified samples will be added to the model.
You want to update your model with a new data, which you have not seen before. Actually, both correctly classified and misclassified unseen samples can improve model performance.
For the update itself, some models are easy to update (Neural Networks - train them several iterations on a new data; k-Nearest Neighbors - just add new data samples to the dataset). Other models are harder to update, for example a linear regression model (check the article "A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks")
sounds like you want to look up some sequential/online learning methods..
http://en.wikipedia.org/wiki/Online_machine_learning
Thank you for the answers @Anfon, @Alex.
I use random forest classifier, 5 labels (classes), sample count is around 2000-3000 and feature vector size is 28.
My learning algorithm is not true on-line because the new samples are not incorporated immediately. New samples arrive, users classify them and upload to the cloud. I download the new samples and it is my decision which samples will be added to the model update to distribute a new model to the users. Thus I don't do real on-line machine learning, but I update the "final" model time to time and distribute back to the users to enhance the prediction power over time.
Mainly, I don't know if there is any specific name for this update style of the final model in the machine learning literature. I can read many things about different classifiers, cross-validation and model selection, but all advice I could find about building a final model that "put all your available data to the final model".
Your approach sounds good to me. I know lots of companies (including some very big ones!) use this approach, of updating their models hourly/daily/weekly etc. Given the size of your model, ie number of features/datapoints you shouldn't have any scalability issues for a while.
One issue I would mention is that your population may not remain stationary over time. For example, the dependency between recently measured features and the classes may be different than those in the ancient past, people and populations can change their preferences over time... Again its probably not something you need to worry about, unless you expect such strong changes in behaviour over time within your application.
- Badges
- Science topic