Hello Sreemanee -

In forecasting, there can be events which cause discontinuities.  Similarly, if you have a model applied cross-sectionally, as it seems, I think, that perhaps you do, then you are still interested in what influences may have newly become felt.  Therefore, though it may be good to have a regular time schedule for review, it may be much more important to review your model whenever something has changed which may impact it.  This requires an exercise of subject matter expertise.  Even a scheduled review process should be based on subject matter knowledge.

Another thought: Although using too many variables can substantially inflate variance, I found, in a fairly simple case for producing periodic reports on official energy statistics, that when the most relevant variable changed from time to time, it worked very well to include more candidate variables.  I am sure there is a limit to this, and it may not work well for you, but you might experiment with that. 

I'm not certain how much of the above may or may not be of use to you, for your particular needs. 

Cheers - Jim

Hi Sreemanee,

This can be considered in terms of the "volatility" of the domain environment.

The % change of statistics of key variables over time (e.g. mean, standard deviation) and their pairwise correlations can be evaluated and quantified to get a "ball park figure.

However, I would say the best option is to have a quality tolerance limit (e.g. plus or minus x) for the value you are predicting and if the precision of the model goes outside  that range then this triggers a retrain. In order to quantify the precision you need a "benchmark" data-set of representative cases where the outcome is known.

Actually this issue is considered in Chapter 2 of my book:

Nettleton, D.F., Commercial Data Mining - Processing, Analysis and Modeling for Predictive Analytics Projects, 1st Edition (2014), Morgan Kaufmann Publishers, ISBN 9780124166028, 304 pages.

Hope this helps !

Best regards,

David.

It sounds as if David is suggesting some version of a Shewhart/control chart??  This might be helpful, after the fact, to let you know if something has 'gone wrong' ... or perhaps is 'going wrong.' 

Thank you James and David for your responses. Thank you also for your suggestions and useful references. I understand that models like the QRISK score is updated annually to account for changes in smoking and obesity rates. Increasingly I am getting interested in stream mining algorithms that get updated as newer data gets exposed to the model affording real-time updating. Wondering if you have any experience in that area. 

Thanks again. 

Not really my area, but updating with the latest information as often as is practical, and can be done accurately, sounds important.