Sökning: onr:"swepub:oai:DiVA.org:hh-51402" >
Continuous transfer...
Continuous transfer of neural network representational similarity for incremental learning
-
- Tian, Songsong (författare)
- Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
-
- Li, Weijun (författare)
- Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
-
- Ning, Xin (författare)
- Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; Zhongke Ruitu Technology Co., Ltd, Beijing, China
-
visa fler...
-
- Ran, Hang (författare)
- Chinese Academy of Sciences, Beijing, China
-
- Qin, Hong (författare)
- Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
-
- Tiwari, Prayag, 1991- (författare)
- Högskolan i Halmstad,Akademin för informationsteknologi
-
visa färre...
-
Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; Zhongke Ruitu Technology Co, Ltd, Beijing, China (creator_code:org_t)
- Amsterdam : Elsevier, 2023
- 2023
- Engelska.
-
Ingår i: Neurocomputing. - Amsterdam : Elsevier. - 0925-2312 .- 1872-8286. ; 545
- Relaterad länk:
-
https://urn.kb.se/re...
-
visa fler...
-
https://doi.org/10.1...
-
visa färre...
Abstract
Ämnesord
Stäng
- The incremental learning paradigm in machine learning has consistently been a focus of academic research. It is similar to the way in which biological systems learn, and reduces energy consumption by avoiding excessive retraining. Existing studies utilize the powerful feature extraction capabilities of pre-trained models to address incremental learning, but there remains a problem of insufficient utilization of neural network feature knowledge. To address this issue, this paper proposes a novel method called Pre-trained Model Knowledge Distillation (PMKD) which combines knowledge distillation of neural network representations and replay. This paper designs a loss function based on centered kernel alignment to transfer neural network representations knowledge from the pre-trained model to the incremental model layer-by-layer. Additionally, the use of memory buffer for Dark Experience Replay helps the model retain past knowledge better. Experiments show that PMKD achieved superior performance on various datasets and different buffer sizes. Compared to other methods, our class incremental learning accuracy reached the best performance. The open-source code is published athttps://github.com/TianSongS/PMKD-IL. © 2023 The Author(s)
Ämnesord
- NATURVETENSKAP -- Data- och informationsvetenskap -- Språkteknologi (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences -- Language Technology (hsv//eng)
Nyckelord
- Incremental learning
- Knowledge distillation
- Neural network representation
- Pre-trained model
Publikations- och innehållstyp
- ref (ämneskategori)
- art (ämneskategori)
Hitta via bibliotek
Till lärosätets databas