Dear David F. Nettleton,

You may find your answers from the papers below.

Alpaydin, E. (1994). GAL: Networks that grow when they learn and shrink when they forget. International journal of pattern recognition and Artificial Intelligence, 8(01), 391-414.

Lu, J., Behbood, V., Hao, P., Zuo, H., Xue, S., & Zhang, G. (2015). Transfer learning using computational intelligence: a survey. Knowledge-Based Systems, 80, 14-23.

Thanks,

Sobhan

Yes, this problem is explored in transfer learning.

https://en.wikipedia.org/wiki/Transfer_learning

Maybe we go further into

Multi-task learning

Dear David F. Nettleton,

You may find your answers from the papers below.

Alpaydin, E. (1994). GAL: Networks that grow when they learn and shrink when they forget. International journal of pattern recognition and Artificial Intelligence, 8(01), 391-414.

Lu, J., Behbood, V., Hao, P., Zuo, H., Xue, S., & Zhang, G. (2015). Transfer learning using computational intelligence: a survey. Knowledge-Based Systems, 80, 14-23.

Thanks,

Sobhan

Let's be fair. There are probably no "life-long" learners which keep their findings and enrich their knowledge by further learning in different domains. Today's method like transfer learning concentrate on models working in similar domains, they do not concentrate on knowledge storage and sharing. You can use layers made for French to translate Romanian, but we are still far from recognizing relations between persons on an image with help from e.g. learning from texts, videos, voice recognition, etc. Many people showed it was possible, but continuous and organized "remembering" doesn't (AFAIK) exist and "shared knowledge" is "forgotten" as soon as experiments are finished...

Dear Turki,

Thanks a lot for this suggestion about transfer learning, from it I found this paper:

Robert M. French, "Catastrophic forgetting in connectionist networks"

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.480.7627&rep=rep1&type=pdf

which seems to be close to the problem idea I had in mind.

However, some current "transfer learning" research seems to be more like making analogies between similar domains, or pretraining a NN with a more general domain which can then be "tuned" for more specific sub-topics within this general domain.

I found this Matlab solution which claims to do this:

https://uk.mathworks.com/videos/deep-learning-with-matlab-transfer-learning-in-10-lines-of-matlab-code-1487714838381.html?s_v1=24391&elqem=2512527_EM_WW_18-07_NEWSLETTER_CG-DIGEST&elqTrackId=b4dc40d4537841dfbaf356a5fb2a0384&elq=5d9bf8253e3c4cd2b69509c3c1d4ee24&elqaid=24391&elqat=1&elqCampaignId=8304

Thanks and regards,

David.

Dear Sobhan and Rafal,

Thanks for your suggested papers and discussion. In the case of language learners, I am not a linguist but it may be interesting to contrast how the biological brain works when learning a similar language to one already known (e.g. Italian already knowing Spanish) compared to learning a language totally different to known ones (e.g. Chinese when knowing English).

There may be some base concepts (nouns, verbs) which are abstracted out as "meta-knowledge" which are then used to access different parts of the overall neural network. So "Chinese" could stored in neural sub-network A and "English" in neural sub-network B, the inter sub-network connections being sparser and the intra sub-network connections being denser.

Thanks and regards,

David.