In the paper "on-the-fly learning with perpetual learning machines" a novel type of DNN is presented - a perpetual learning machine. The perpetual learning machine is made up of two DNNs - one for identification, and one for recall. Simpson trains the model on the MNIST data set "We took the first 75 of the MNIST digits and assigned each to an arbitrary class." The recall DNN is used to feed images to the storage DNN, alowing the model to perform self-supervised training in 'perpetual stochastic gradient descent'.
My question/ concern is that each class is only made up of one image, so how can this model be applied when a class is made up of 1000's of examples. How can the recall DNN synthesize a an image representative of all/ any instance of a given class? Another question that arises, is why dropout and
dither are used - over-fitting cannot be an issue if the training set is the same as the evaluation/ test set?
Thanks in advance for your time, I understand that this is a very novel area, and prehaps I am mistaken as to what the author meant.
https://arxiv.org/abs/1509.00913
Hi Monyer,
Thanks for your contribution. Unfortunately I think the paper you have linked is not relevant. My question pertains to Perpetual Learning Machines which is a specific network architecture proposed by Simson in 2015. The defining feature of these systems being the presence of 'storage' and 'recall' networks. Since the paper you linked was published before the advent of the modern neural network (see Alexnet 2012), I don't see how it is relevant. Feel free to elaborate though!
https://www.cs.tau.ac.il/~dcor/articles/2016/Holistic.pdf
may be useful
.
when a class is composed by more than one instance, i would just draw one instance at random for each sgd iteration and train the networks on the (class, instance) pair
in the set up described in the paper, there is indeed a training set and a test set (the terminology is not very adequate for on the fly learning) as the first 50 instances are not used on-the-fly, only in the initial training phase (later on, they are just "remembered") ; the author points out that the absence of regularization leads to an explosion of the error during the perpetual sgd phase but does not comment on the possible reasons for this behaviour (specifically, he does not indicate if the error explosion is for all instances or just the instance being only remembered)
.
https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence