There is a general tendency not to use pre-computed features with DNN's, since researchers expect the network to learn the relevant features themselves. However, the real deep neural network (the brain) does use a number of hard-coded features. For example, the basilar membrane in the cochlea performs subband decomposition like feature extraction. Hubel and Wiesel found that there are dedicated neurons that are sensitive to edges in specific directions. So, shouldn't we think that there is a definite case for hard coded feature extraction, even with the use of DNN's ?
A neural network consisting of 'neurons' is a mathematical abstraction of a biological entity. Thus, there are significant differences in the way the hard coded features are extracted. In the human brain, the neurons for extracting those hard coded features are highly specialized, indicating the need for specialized 'artificial neurons' for performing the equivalent task.
There may be a case for this, but as Nahian stated, these features are highly specialized in the brain. There would need to be studies and research on how to best identify and craft these few, specialized hard coded features. One of the issues I see with this is that I tend to think many researchers would assign these features arbitrarily. Which would be at odds with biological intention, and also would be counterproductive to the nature of neural networks. Also, these tasks that we use neural networks are very specific, compared to the ones in the brain that are multifaceted. So identifying the RIGHT type of hard coded features for your task could become too much of a burden.
It is an interesting idea to explore in theory though.
I would say that generally the lower-level features (learned by the first few conv layer) are highly transferable, but the higher-level are more task specific, so it is reasonable to transfer them only if your task is closely related to the task for which the CNN was trained for. There is definitely not a binary amswer for this question.
Thinking about representations will always be essential in the design of machine-learning tools.
First: Not all applications are image oriented. Whereas the convolutional square patch is meaningful both in regular image processing and image classification using CNNs, this does not generalize to other data types such as industrial process time-series data ('.csv' spreadsheets) with arbitrary ordering of columns. Averaging over columns is meaningless unless a natural ordering of dimensions over columns has been applied.
Second, thinking about representations in a deep-learning context has been shown to be highly successful in attribute learning, where a high-dimensional binary target vector is used. Many applications would not work with one-hot representation (cf. Chinese characters: 106k classes in the output layer? Really?). The attribute output representation with a few hundred dimensions can be reused and allows single- or few-shot learning, with a variable number of classes over time, using kNN or SVM, as opposed to the CNN as a whole. The latter would require a retraining of at least the last layer. As argued above, this is senseless for problems with massive numbers of classes.
Thirdly, some non-linear operations are maybe realizable with, say, 2-3 layers in a neural network, but why wait until a neural network learns 'Otsu' on the input or on a feature map of a CNN, if that can be computed directly from intensity variances?
Fourth, the magical goal of 'end to end' is highly misleading. Rather than constituting the complete path of sensor to effector, the input of current common benchmarks is often highly contrived, normalized, cropped, often using highly distorted aspect ratios of common photos (cf: giraffes with short thick necks).
Such data sets are also often meticulously curated. Experience in our own lab has shown that there is no low-hanging fruit when applying, e.g., 'Inception V3' on a metal-defect problem in industry. In character recognition, a CNN method which is demonstrably strong in nicely segmented characters may fail terribly, if the CNN is _really_ applied in a convolutional manner, such that the complete input patch is moving as a window over continuous text. The network will see intermediate shapes that were never met during the sterile training setup, with a subsequent drastic drop in effective performance.
Finally, biological systems indeed show a host of useful representations: Forgive me for plugging my own earlier work:
http://www.ai.rug.nl/~lambert/papers/ieee-smc-2004-schomaker.pdf
Dear Prof. Schomaker,
I am delighted to see your inputs and my heartfelt thanks for the same. Many thanks also for giving the link to your relevant publication on the same. I shall come back to you after reading the same.
Dear Prof. Schomaker, I enjoyed reading your SMC paper. In fact, I was exactly looking for such a paper, not only for learning, but also to use as a reference to justify what we are doing to improve the recognition performance of a network.
- Badges
- Science topic