Interpretability is closely connected with the ability of users to understand the model. Typical criteria are:

* a small number of input features (only the necessary ones), ideally not more than 2-3 to make a simple visualization possible,

* features fitting to user's expectation (e.g., based on a priori knowledge that can be formalized as a priori relevance),

* a simple internal processing, e.g. by understandable rules instead of a complex processing inside an Artificial Neural Network.

It depends on the domain and the application, but I would say that it is usually quite important. Some models are 'black box' in a sense that data comes in and data comes out but we can not really interpret the model once it's learned. Sometimes this is sufficient, if the only thing we would be interested in is to make an effective automated prediction system. However, our models often reveal new hidden patterns in the data and act as a part of a positive feedback loop back to the domain experts. The experts can indeed learn new dependencies and correlations from those models that can be easily interpreted, like rule-based systems or trees or graphical models. Let's say we're trying to predict the effectiveness of a certain drug in its therapeutic use, based on each patient's history and condition. Even though a good predictor would certainly be useful in practice, making a model that reveals the reasons why the drug would or would not work in specific cases would be much more meaningful and would enable the experts to design better therapeutic drugs in the future. There are many examples like this - and we must not forget that computer science is a Science and not just engineering - the purpose is not only to build effective components, but also to learn new things and increase our knowledge of the world we live in.

In the context of machine learning, interpretability helps us understand how the model has learned to solve (or tried to solve) a given problem.

I think that a models most important qualities is its precision. If the model is precise based on the assumptions made, then you can either:

1) Be able to make predictions

2) Explain the phenomena being modeled

These two qualities would make the dependencies for my interpretation of the model.

Based on this definition of interpretability (which may not be the only one) the importance to data mining with regards to the first point is :can the trained algorithm be used to make other predictions besides the one on the given datsets? This goes more to the generalization power of the algorithm and the qualities of the dataset for the specific problem being modeled

The second point is more subtle and requires more thought on how does the result of a data mining experiment helps in explaining an underlying phenomenon. While Nenad's post is right on with regards to rule finding for drugs(which would make for better applications) I would add a more philosophical question: Can a data mining technique based on data replace simpler rules of modeling based on closed form equations which provide higher level abstractions that would not fit any real world data (unless you eliminate all outliers to have a clean line, etc.).

My experience is that we are relying more and more on data mining to derive concrete models rather than abstract those models into closed mathematical equations. I think that as we derive precision (complexity) from data in the model we are loosing predictive capacity. I would rather see a more balanced approach were we use data mining techniques to derive initial results on which to feed our intuition then take it a step further and derive more abstract mathematical models.

This is just my biased opinion

The judge of interpretability is the user of the model. I work with the marketing department of my compagny. Usually, they like to know some caracteristics of customers to target marketing campaigns. Here, the interpretability of a model can help. If you know that a customer is going to churn, it can help your targeting. But, if you know why a customer is going to churn, you can propose something to this customer in order to improve his loyalty.

An important point: interpretability and accuracy are usually concurrent tasks:

to interpret a classification it is important to use a few number of variables, but if you need accuracy, you have to use a lot of variables. My opinion is that you need two models: one for prediction and another one for interpretation.

I developped a methodology to explain Neural Network classification (see https://www.researchgate.net/publication/11348549_A_methodology_to_explain_neural_network_classification). It consists in a feature selection step and then an interpretation step using a few number of selected features. The interpretation is based on the causal importance. This notion is based on "what if similations". For example a variable that can change the class of an instance has an high causal importance.

Article A methodology to explain neural network classification

I worked on rule extraction from single neural networks and ensembles of neural networks. There is a clear tradeoff between model complexity and model interpretability. For instance, a boosted ensemble of neural networks can be more accurate at the cost of more extracted rules and more conditions per rule. A bagged ensemble of neural networks is very often slightly more accurate than a single model with also very often lower rule complexity. See For instance my downloadable publications in ResearchGate :

1. A model for single and multiple knowledge based networks.

2. A study on rule extraction from several combined neural networks.

Hi all,

To fuel the discussion from the engineering design theory and methodology side. In context of research efforts in the area of model-based engineering design, the model interpretability, let’s say of a technical system model, can be viewed as a model’s property to be successfully exchanged and understood among different disciplines and various design teams. In a formal sense the existence of translation within different contexts is what matters. Implications are many – ranging from content loss, e.g. accuracy reduction, to misinterpretation.

As Nenad Tomosev says, interpretability in models depends on the domain and the application like in Precision models,Machine Learning,Decision Support,Semantic Models,Visualizable models etc.

In this context, earlier Discussions made on " How to make Prediction Models more Inetrpretable " ,given in the link below, may be of some interest.

http://stats.stackexchange.com/questions/38436/how-to-make-prediction-models-more-interpretable

Interpretation of Models are generally used for finding the response of a system,with the given inputs and constraints and study their output.Based on the output we fine tune the model or change in inputs or constraints by observing the output and interpreting the effect on the components inputs,controls etc every time the model is redefined, starting with a very simple model. This Philosophy applies for modeling of any system.

P.S.

Interpretability is closely connected with the ability of users to understand the model. Typical criteria are:

* a small number of input features (only the necessary ones), ideally not more than 2-3 to make a simple visualization possible,

* features fitting to user's expectation (e.g., based on a priori knowledge that can be formalized as a priori relevance),

* a simple internal processing, e.g. by understandable rules instead of a complex processing inside an Artificial Neural Network.

The topic is very interesting and it has been raised so many times in the two years after the question was posted in researchergate.

A few days ago (April 2016) a post on the KD nuggets website discussed "the myth of model interpretability"

http://www.kdnuggets.com/2015/04/model-interpretability-neural-networks-deep-learning.html

while  in June 2016 there will be a workshop on Human Interpretability in ML at the ICML conference 

https://sites.google.com/site/2016whi/

http://www.kdnuggets.com/2015/04/model-interpretability-neural-networks-deep-learning.html

https://sites.google.com/site/2016whi/

Hi, if you find blog post interesting (I authored it), you might be interested to check out the longer, more formal and developed version (http://zacklipton.com/media/papers/mythos-model-interpretability_16.pdf) that  will be published at the workshop at ICML.

I believe the reason why the model came up is also important for users, especially for decision makers in data mining domain. For example, diaper and beer should put together is a simple rule, but without knowing what makes this rule exist, decision makers can't use it in reality.

I think this is a fantastic question and it's one that I am dealing with in my research. In the social and behavioral sciences, this attribute (at least currently) is essential for researchers, policymakers, among others. In fact, this attribute is often more sought after than predictive accuracy. The lack of interpretability, in terms of understanding relationships among variables (or why the model is accurate), is one of the reasons data mining and machine learning techniques aren't openly embraced in my field.

In the last couple of years, the ICML conference hosted the Workshop on Human Interpretability in Machine Learning in order to tackle exactly your question. As you can see from the papers (see links below), the issue is still open (and I even doubt that some of the papers present something "interpretable" at all).

It seems that rule based approaches are easier in this sense (I just read a paper on the JMLR on this topic).

A few years ago, I found a paper about the interpretability of neural networks interesting.

Probably, if you can find a clever visualisation approach you may get to the point where the output (and the model) becomes easier to understand.

In this respect, the presentation of the parallel coordinates by Inselberg is a small gem.

https://sites.google.com/site/2016whi/

https://sites.google.com/view/whi2017/home

http://jmlr.org/papers/volume18/16-003/16-003.pdf

http://www.sciencedirect.com/science/article/pii/S0893608001001277

https://www.microsoft.com/en-us/research/video/parallel-coordinates-visual-multidimensional-geometry-and-its-applications/

Is there any evidence to suggest that interpretability is becoming more important again and recently after taking a back seat in the last few years while the majority of machine learning research has focused on techniques for generating more and more accurate models? Have phenomena such as Kaggle competitions increased the hype for accuracy, while practical applications with an emphasis on knowledge discovery been sidelined?

Julian,

I would answer your question negatively. According to the Gartner's peak of inflated expectations

( https://www.gartner.com/smarterwithgartner/top-trends-in-the-gartner-hype-cycle-for-emerging-technologies-2017/ )

Machine learning is still at the peak and so is deep learning. Once the hype subsides, then noise will subside. This will allow the researchers to promote the sobering structure that is needed to continue developing the field without these distractions.

They refere to approaches that after learning it is posible to know the knowledge