In machine learning, the trained model is sometimes too complicated thus it may be hard to run the model due to resource limitation (e.g., mobile devices). Some research proposes to use other models to approximate the complicated ones. However, my question is how can we make sure the approximation makes sense?
Reference:
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006, August). Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 535-541). ACM.
one measure is the cross validation performance (such as the ROC) of the simplified model versus the original
Testing predictive ability is one way to judge the performance of approximate model. Put labelled data (which was not used to build the model) into the model and test the rate of correct classification. There are several measures (depending on the data structure) such as ROC Curve, AUCPrR.
Maybe one way to think about it in theoretical terms is as a projection operation. The original model's functional form constrains what it can learn and - assuming you have fit its parameters to a reasonable result - what you are trying to do is project the learned representation to a different, simpler and thus less expressive class of functions.
This is sure to result in loss of accuracy but the question is whether in the cases you care about i.e. examples drawn from the distribution that you are going to use it in production, that loss is meaningful. Hinton had a really interesting presentation on what he dubbed dark knowledge that might be of interest to you:
http://www.ttic.edu/dl/dark14.pdf
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence