I've read several times that on the problems of large dimension (Image Recognition, Text Mining, ...), the Deep Learning method gives significantly higher accuracy than the "classical" methods (such as SVM, Logistic Regression, etc.). And what happens on problems of ordinary, medium dimension? Let's say that the data set is on the order of 1,000 ... 10,000 objects and the object is characterized by 10 ... 20 parameters. Are there articles that provide data on the comparison of accuracy indicators (Recall, Precision, ...) by Deep Learning and other methods on some benchmarks?
Thanks beforehand for your answer. Regards, Sergey.
Hello, it is a bit hard to answer, and there are many aspects to consider. The standard methods, in general, does not like large feature space and so there are techniques how to decrease the space while keeping the information (PCA, learning manifold, etc.) and some methods also tend to have long learning phase for large datasets depending on its optimization criterion. On the other hand, DL may not be fast for large datasets that is why the batch strategy is used. For DL you also frequently do not think too much about the feature extraction since is somehow part of the learning in convolution layers...
Regarding your question about literature, you just look to one of the first parts about DL where the basic idea is explained, they usually compare themselves with the standard techniques:
* http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
Yeah here accuracy is a flimsy thing !!!!!!!
We try to use the basic : Total - Missed /Total
CNN I think also does this again here dynamics can be modified and it is reflected in accuracy .
Dear Jiří Borovec, thanks for you answer. I will carefully study attached article.
Dear Aparna Sathya Murthy, thanks a lot for your explanation. But, unfortunately, I did not understand - for regular tasks (~ 1000...5000 objects, ~ 5...10 features) could we to attempt to use Deep Learning instead of SVM, or we should not get essentially accurate results?
Regards, Sergey.
First of all to be clear, I call deep learning all the algorithmic advances in the neural networks since 2000s. Today, a simple MLP-like network (input layer->2-3 hidden fully-connected layers->output layer) with the new learning techniques are much more capable to learn than in 90s. I mean there is really a _huge_ gap after the advances.
I can exactly give my own experience with deep learning on small/medium size problems. I was skeptical a few years ago because I worked with classical neural networks for years, but deep learning at least so good on small problems as traditional methods. I worked on problems with sensor data and audio in the past years and deep learning was not a simple drop-in replacement, but it always granted a few percents (2-5 %) of accuracy improvements all the time. And I talk about feeding the same handcrafted feature vectors to the neural nets, nothing fancy work with autoencoder layers or so. My problems involved 5.000-1.000.000 training samples, 4-14 classes and feature vectors with 10-100 real values. My network topology involved only 2-4 fully-connected layers with 20-100 neurons between the input and output layers.
The best thing in smaller data dimensions (e.g. sensors or audio) that you get all the advantages of the deep learning algorithms, but you can train the small neural topology (model) on CPU or a lowend GPU in reasonable short time, you don't need to invest heavy GPU machinery or cloud services.
I also argue the popular belief that you need a huge dataset with deep learning. I just recently trained a simple CNN model to detect clear sky with my cheapo Raspberry Pi camera in the night to know when I should go out for stargazing on my balcony. I could build a model with ~3000 raw, low-resolution images which was able to generalize/handle all disturbing conditions (different ice patterns on the covering plastic dome, Moon on the sky).
Dear Sergey Porotsky,
Thanks for asking a good question.
I would like to add that deep learning did both the task (feature extraction and classification) together as also mentioned by Jiří Borovec.
The deep learning generally used neural networks with several layers with large number of neurons. Each neuron learns a different basis function. An increase in the layers/neurons also increase the capacity of the network (not accuracy). To generalize these large networks a significantly large number of training objects are required (atleast 10 times the number of parameters in a network) otherwise it will simply over fit even with regularization constraints.
However, you problem is quit small (~ 1000...10000 objects, ~ 10...20 features) for this MLP, SVM (kernel), Logistic Regression, etc. may gives better performance. I don't think that this problem is suitable for the deep learning.
The deep learning (such as CNN) is widely used in the image classification but there is also a good strategy is the manually visualization the feature maps and the filters and confirms that weight are different and the feature maps are leading to some kind of semantic interpretation according to the task. Otherwise a deep learning model with increased capacity may learn the discrimination based on the irrelevant sections of the images (e.g., background, noise, etc.).
Regards,
Dinesh
Dear Csaba Kertész , thanks a lot for your detailed answer. I fully agree with you, that for special tasks (e.g., image and audio recognition) Deep Learning gets better results. But what is your opinion - for regular tasks (~ 1000...5000 objects, ~ 5...10 features) could we to attempt to use Deep Learning instead of SVM, or we should not get essentially accurate results?
Regards, Sergey.
Dear Dinesh Singh, thank your very much for your explanation. Regards, Sergey.
Deep learning is a way of having several transformations in a row, to combine layers and layers of features. SVMs typically only allow a single transformation. Neural networks allow dozens (to hundreds in the latest papers) of layers. Kernel SVMs are like a 2 layer neural network.
This is why deep neural networks, aka deep learning, are currently beating everyone. You can also put a (kernel) SVM on top of a deep neural network. But most of the time, people use a softmax regression on top of deep neural networks.
By deep learning, most people mean "deep neural networks". But other methods of machine learning are trying to become "deep" too
While in case of logistic regression and SVM, the main difference is that logistic regression can only separate linearly separable classes where as SVM (with the kernel trick) can find any arbitrarily shaped decision boundary. This means that SVM will usually do better separating your classes (at least on your training set) but is more prone to over-fitting.
Linear regression is also a simpler model with fewer hyper-parameters to tune (zero if you're not using regularization) making it easier to implement.
Unless you have very good intuitions about the separability of your data, I would suggest start by fitting a logistic regression and if it isn't giving you satisfactory class separability, then try an SVM.
On the other hand, if you find that SVM is over-fitting no matter how you tune the hyper-parameters, consider trying logistic regression.
One final point - logistic regression outputs a probability of being in the positive class (you still need to choose a threshold to make it a classifer), SVM just outputs the classes. SVM can give you probabilies via Platt scaling but this can be very slow.
you may try all of them , and chose what is best for your solutions .It is hard to say what will be the best solution for your problem . since there are lots of factors that can effect the performance of your networks. When you go deep , it means you have lots of choices to do so , and chose the one which is best for you.
- Badges
- Science topic