Abbas Cheddad

I would add to your comment that when the liability of a conditional risk associated with action is high as with a false positive in the medicine field (as you have pointed), then more "transparent" algorithms will be chosen.

Arturo Geigel

I couldn't agree more. I'm glad to hear your point of view. Thank you!

I guess, nowadays, scarcity of data is not an obstacle. In fact, current algorithms are struggling to efficiently scale-up in the era of big data and cloud computing. That is just my opinion and thank you for your input!

Depending on the type of handcrafted feature, the ideas behind some of the handcrafted features may be built into deep networks going forward, resulting in a synergy between the two.

Shounak Datta Thank you!

Don’t you think that feeding forward our own shallow features to the deep network negates the very essence of deep learning concept?

Plus, if that is feasible, how could it be done (in image context)?

There are many questions around this topic, which is why I opened this discussion.

Feeding the shallow features through a deep network would defeat the whole purpose of deep learning. What I meant was, it might be possible to make parametric assumptions about the deep network based on insights from these classical features. Not clear how exactly to do that because it is possibly quite non-trivial. If possible, then making such assumptions could bring down the number of tunable parameters in the deep networks.

Theoretically plausible and it could result in reduction in the learning phase but practically am not sure how that could be implemented nor how it would behave.

Thanks for the input!

Shounak Datta , the challenge with your suggestion lies in the attribution problem of learning algorithms.Current algorithms penalize error homogeneously or randomly adjust weights. Your suggestion would imply discriminately targetting certain neurons to force them to learn patterns. The selectivity of neurons would slow down the learning process substantially but could result in more "targetted " networks

Arturo Geigel

I see, but shouldn't targeting certain neurons to force them to learn patterns that are presumably robust and highly descriptive (handcrafted features) would lower the learning process?

Abbas Cheddad

You are forcing the network to learn in a particular way and this will limit its generalization capability at the expense of getting some targetted response from the network.

The challenge is not as much on the impact to the learning process, because they will learn (at least something). The real challenge is how to determine which nodes to target to produce the desired effect. As the representation gets more distributed (along the nodes of a layer and across layers) this becomes even more difficult.

My view on learned features and how handcrafted (or more general prior information) could make it better.

Deep learning is data and computation hungry. The way of learning is still sensitive to hyper parameter and gradent descent is strapped at local optimal.

In a perfect world, we have infinitive data, infinitive power and infinity depth and complex network, so it can generalized the problem very well.

However, in reality, we have limited computing power, could not test all possible combination of hyperparameters, could not train multiple times/learning rate to get better local optima. So, what is the problem? we are giving them too non-linear problem, too difficult problem so that they could not converge with limited power and computing resource.

In this regards, handcrafted features will be useful. It will help to limited the solution space, to reduce number of thousand weights just to match the handcrafted features. On the other hand, the network itself can force to learn something else useful beside the handcrafted features. (somehow sound similar to residual learning).

Another aspect is deep learning is generalized features from its dataset. So it ends up with general good performance. If your input image is very unique, special pattern, the DL based approach might fail or not as good as handcraft features. It is because, the algorithm depend to much on external information (which is generalized from the training dataset) so that internal information is not take care very much. In this regards, combining internal and external information would boost the robustness of the network.

In conclusion, I expect handcraft feature would be useful to reduce complexity of network, or mixed with learned feature to delivery a better results with lower complexity.

Thuong Nguyen Canh ,

Totally agree on your point, though I think that the problem that Shounak Datta referred to as "based on insights from these classical features" can only be taken so far in reproduction based only on data. While the algorithm can "mimick" the behaviour, so far as I know besides statistical correlation in performance there is no hard guarantee (proof that the internal representations are carrying out the same operations) that the deep learning network will do exactly what the classical algorithm is doing. That is why I think that while statistical algorithms are needed there are also more traditional approaches that have high bias but can be analysed for causality and add them to the NN's toolkit.

This goes back to my original post that "when the liability of a conditional risk associated with action is high" would you go only on statistical correlation as opposed to theoretical underpinnings based on causality?

Arturo Geigel, the parametric assumptions I talked about need not be enforced the way you're envisioning it. Yes, if one were to do it the way you suggest, there indeed be no way to control what the network learns, giving rise to problems with generalizability among other things.

Shounak Datta what alternative method of enforcement do you suggest?

The closest that I can come up with resembling your suggestion is a variational auto encoder that maps latent variables, but even if we somehow doctored the dataset there is no guarantee that the latent variable mapping will correspond to our underlying assumptions.

Let me know

Arturo Geigel, there may be other ways to do this. For example, a naive way would be to have convolutional filters which are tailored to detect certain patterns. Like I said, it is not exactly non-trivial to do. Hence, there is a need to think deeply.

Shounak Datta ,

Wouldn't that imply "discriminately targetting certain neurons to force them to learn patterns"?

Arturo Geigel not in the sense of "penaliz[-ing] error [non-]homogeneously". If you think about it, much of what we currently do with deep learning can also fall under the purview of "discriminately targetting certain neurons to force them to learn patterns". For example, so much as assuming a prior on the input or output distribution can cause such 'discrimination'.

Another way I would go for the hybrid approach. We force the network to learn something else that useful beside the well designed patterns. So that all parameters are used to compensate whatever missing in the well designed patterns. Somehow that mimics the orthogonal property.

Shounak Datta ,

I would suggest that, whether one assumes a prior or make it into a convolutional layer, the network has to learn. If one takes learning as working by stimulation or lack thereof as definition. One can also say as well that stimulation is simply reward and lack thereof is penalization, especially with regularization techniques (You can disagree with me on this one for sure, and I apologize for being a simple minded binary logical thinking unit).

I agree with you that much of the techniques can be considered discriminately targeting certain neurons. But, I would disagree in "assuming a prior on the input or output distribution" to be part of the learning since it is being "inserted artificially into the process. On the convolutional NN, I would also make the same note depending on how it is 'tailored'.

While it is the mainstream trend to take NN optimization regardless of biological plausibility, I still maintain this distinction of what is part of the learning algorithm and what is not (I apologize again for hair splitting).

Arturo Geigel, so then you agree that "discriminately targetting certain neurons to force them to learn patterns" need not lead to loss of generalizability? In fact, for models as complex as deep networks, that might actually lend more generalizability.

Shounak Datta ,

To see my point on ‘discriminately targeting certain neurons to force them to learn patterns" need not lead to loss of generalizability’ we need to shift from looking at the structure of your argument to the structure of my argument to see where my objections and caveats lie (I have listed the references not to be pedantic but so that you can trace the origin of the argument to see if it is complete or revisions are warranted).

Here it goes

Let us take as starting points [1] and [3] where both agree on the representational power (i.e. approximation accuracy based on the number of neurons). This also, concurs with the work in the 80’s where part of the focus of study was on the capacity of neurons in terms of information theory. One might at first infer from this that as we reduce the number of neurons the more “rough” the approximation and therefore generality. While this may be a necessary condition to state generalization, in my view it does not comply on being both necessary and sufficient to establish generalization.

Let us see what I take to be sufficient (I take that the above can establish necessity).

I will use the argument in [2,4] and [5] to make the link from neural networks to PCA. The gist of the argument is that if you restrict the number of nodes to get the 3 principal components and you incur in “discriminately targeting certain neurons to force them to learn patterns” outside the main 3 principal components, what guarantees do you have that the targeted nodes have not blown away the information required for generalization on these three principal components. This would be my sufficiency argument, that you need at least those nodes intact to model that exact desired level of generalization. If this condition is not met one cannot not guarantee generalization(in this sense not just any generalization, but the desired one).

You can overcome this by adding capacity to the neural network but that is not part of the learning (deep or otherwise). If I grant a yes to the statement, it would only be under the assumption of a suboptimal NN with excess capacity as a starting assumption. Instead I would focus on making two NN’s that are optimal instead of one suboptimal to begin with.

[1] Hornik, K. (1993). Some new results on neural network approximation. Neural networks, 6(8), 1069-1072.

[2] Oja, E. (1982). Simplified neuron model as a principal component analyzer. Journal of mathematical biology, 15(3), 267-273.

[3]Le Roux, N., & Bengio, Y. (2008). Representational power of restricted Boltzmann machines and deep belief networks. Neural computation, 20(6), 1631-1649.

[4]Sanger, T. D. (1989). Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural networks, 2(6), 459-473.

[5]Wang, Y., Yao, H., & Zhao, S. (2016). Auto-encoder based dimensionality reduction. Neurocomputing, 184, 232-242.

Arturo Geigel,

I don't see how you made a jump from your para 1 and 2 to 'excess capacity'.

But, if I were to take the gist of your concerns to be the fact that putting constraints on the network may lead to loss of information because the model 'chooses' (for want of a better word) to not see the relevant bits, then I would say it is a valid but overtly speculative concern.

One cannot, without making enough investigations, claim that such a model will always 'fail' (as you suggest) or always 'succeed'. One can only try it out and see if it does what we expect it to do (the point I've been making).

Shounak Datta ,

1 is capacity->representation

2 is representation->capacity

If 2 is not given bounds for success, then I have to start under the assumption that it will fail and find instances in which it does not and explain why (it is not just making it work to be science). To this day, I have not found how to get clear bounds for the problems of multiple feature extraction under features that differ in bias/variance requirements (rejecting this as a starting point would be such a strong assumption). Another assumption that would be needed to even contemplate 2 as a possibility is feature complexity and similarity.

My view is that so far this is so problematic that in terms of our initial discussion. To solve the above, one would need feature engineering or restricting the data given to the neural network as a learner. Under this scenario, I would not see much of a difference between a "raw" neural network (without all this additional help) and handcrafted features and using different neural networks.

Our views on the subject are different in the sense that you are viewing it from an experimental point of view while I am viewing it from a theoretical point of view. There is nothing wrong with this as long as we make it explicit. I understand from an experimental point of view that one can eventually get results (given enough time and money which is also a strong assumption).

One example of what we discussed so far

Preprint Learning Filter Basis for Convolutional Neural Network Compression

Thuong Nguyen Canh, thanks for sharing the paper. EDIT: Unfortunately, looks like that paper is not public anymore.

A quick update from IEEE ICIP 2019, Taipei, Taiwan.

I was lucky to ask Yann Lecun myself about the role of the handcrafted features - which is our discussion so far. According to his answer, the role will be very minor as "why we should use the handcrafted one when we can learn it with a much better representation''.

I also point out the possibility of using the handcrafted feature for "explainable AI". He responded with an example of performance-wise of 80% accuracy, explainable hand-craft feature vs. 95% accuracy learned features, then most likely people will prefer 95% accuracy and tolerate the not explainable part. He used medical drug treatment as the examples that we did not really know whether some drugs are working or not.

But he also stressed the importance of designing dataset and test set so that it does cover the distribution of the problem, and the target that we are aiming at.

Hope it adds something useful.

Thank you Thuong for the update, very good input. But allow me to question his statement that "most likely people will prefer 95% accuracy and tolerate the not explainable part.", well not all. For the example he gave on the binary drugs classification it should not be a problem but in some medical fields, researchers would prefer to have a backward feedback on the importance of certain features so that they can further explore them physiologically (e.g., causality studies).