Table of Links Abstract and 1 Introduction Abstract and 1 Introduction Related works


Problem setting


Methodology
4.1. Decision boundary-aware distillation
4.2. Knowledge consolidation


Experimental results and 5.1. Experiment Setup
5.2. Comparison with SOTA methods
5.3. Ablation study


Conclusion and future work and References Related works Related works Related works Problem setting Problem setting Problem setting Methodology
4.1. Decision boundary-aware distillation
4.2. Knowledge consolidation Methodology Methodology 4.1. Decision boundary-aware distillation 4.1. Decision boundary-aware distillation 4.2. Knowledge consolidation 4.2. Knowledge consolidation Experimental results and 5.1. Experiment Setup
5.2. Comparison with SOTA methods
5.3. Ablation study Experimental results and 5.1. Experiment Setup Experimental results and 5.1. Experiment Setup 5.2. Comparison with SOTA methods 5.2. Comparison with SOTA methods 5.3. Ablation study 5.3. Ablation study Conclusion and future work and References Conclusion and future work and References Conclusion and future work and References Supplementary Material Supplementary Material Details of the theoretical analysis on KCEMA mechanism in IIL
Algorithm overview
Dataset details
Implementation details
Visualization of dusted input images
More experimental results Details of the theoretical analysis on KCEMA mechanism in IIL Details of the theoretical analysis on KCEMA mechanism in IIL Algorithm overview Algorithm overview Dataset details Dataset details Implementation details Implementation details Visualization of dusted input images Visualization of dusted input images More experimental results More experimental results 2. Related works This paper devotes to the instance-incremental learning which is an associated topic to the CIL but seldom investigated. In the following, related topics on class-incremental learning, continual domain adaptation, and methods based on knowledge distillation (KD) are introduced. Class-incremental learning. CIL is proposed to learn new classes without suffering from the notorious catastrophic forgetting problem and is the main topic that most of works focused on in this area. Methods of CIL can be categorized into three types: 1) important weights regularization [1, 10, 19, 32], which constrains the important weights for old tasks and free those unimportant weights for new task. Freezing the weights limits the ability to learn from new data and always lead to a inferior performance on new classes. 2) Rehearsal or pseudo rehearsal method, which stores a small-size of typical exemplars [2, 4, 9, 22] or relies on a generation network to produce old data [23] for old knowledge retaining. Usually, these methods utilize knowledge distillation and perform over the weight regularization method. Although the prototypes of old classes are efficacy in preserving knowledge, they are unable to promote the model’s performance on hard samples, which is always a problem in real deployment. 3) Dynamic network architecture based method [8, 15, 30, 31], which adaptively expenses the network each time for new knowledge learning. However, deploying a changing neural model in real scenarios is troublesome, especially when it goes too big. Although most CIL methods have strong ability in learning new classes, few of them can be directly utilized in the new IIL setting in our test. The reason is that performance promotion on old classes is less emphasized in CIL. Class-incremental learning Although the prototypes of old classes are efficacy in preserving knowledge, they are unable to promote the model’s performance on hard samples, which is always a problem in real deployment. Knowledge distillation-based incremental learning. Most of existing incremental learning works utilize knowledge distillation (KD) to mitigate catastrophic forgetting. LwF [12] is one of the earliest approaches that constrains the prediction of new data through KD. iCarl [22] and many other methods distill knowledge on preserved exemplars to free the learning capability on new data. Zhai et al. [33] and Zhang et al. [34] exploit distillation with augmented data and unlabeled auxiliary data at negligible cost. Different from above distillation at label level, Kang et al. [9] and Douillard [4] proposed to distill knowledge at feature level for CIL. Compared to the aforementioned researches, the proposed decision boundary-aware distillation requires no access to old exemplars and is simple but effective in learning new as well as retaining the old knowledge. Knowledge distillation-based incremental learning. Comparison with the CDA and ISL. Rencently, some work of continual domain adptation (CDA) [7, 21, 27] and incremental subpopulation learning (ISL) [13] is proposed and has high similarity with the IIL setting. All of the three settings have a fixed label space. The CDA focus on solving the visual domain variations such as illumination and background. ISL is a specific case of CDA and pays more attention to the subcategories within a class, such as Poodles and Terriers. Compared to them, IIL is a more general setting where the concept drift is not limited to the domain shift in CDA or subpopulation shifting problem in ISL. More importantly, the new IIL not only aims to retain the performance but also has to promote the generalization with several new observations in the whole data space. Comparison with the CDA and ISL. Authors:
(1) Qiang Nie, Hong Kong University of Science and Technology (Guangzhou);
(2) Weifu Fu, Tencent Youtu Lab;
(3) Yuhuan Lin, Tencent Youtu Lab;
(4) Jialin Li, Tencent Youtu Lab;
(5) Yifeng Zhou, Tencent Youtu Lab;
(6) Yong Liu, Tencent Youtu Lab;
(7) Qiang Nie, Hong Kong University of Science and Technology (Guangzhou);
(8) Chengjie Wang, Tencent Youtu Lab. Authors: Authors: (1) Qiang Nie, Hong Kong University of Science and Technology (Guangzhou); (2) Weifu Fu, Tencent Youtu Lab; (3) Yuhuan Lin, Tencent Youtu Lab; (4) Jialin Li, Tencent Youtu Lab; (5) Yifeng Zhou, Tencent Youtu Lab; (6) Yong Liu, Tencent Youtu Lab; (7) Qiang Nie, Hong Kong University of Science and Technology (Guangzhou); (8) Chengjie Wang, Tencent Youtu Lab. This paper is available on arxiv under CC BY-NC-ND 4.0 Deed (Attribution-Noncommercial-Noderivs 4.0 International) license. This paper is available on arxiv under CC BY-NC-ND 4.0 Deed (Attribution-Noncommercial-Noderivs 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Incremental Learning: Comparing Methods for Catastrophic Forgetting and Model Promotion

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Coarse-to-Fine Approach to Text-Guided 3D Localization Without Ground Truth

Understanding GAN Mode Collapse: Causes and Solutions

Data Scarcity Solution: S-CycleGAN for CT-to-Ultrasound Translation

Optimizing SAGE Net: Achieving High Performance with Shorter Input Sequences for Online Inference

Model Promotion: Using EMA to Balance Learning and Forgetting in IIL

Evaluating Instance-Incremental Learning: CIL Methods on Cifar-100 and ImageNet

A Coarse-to-Fine Approach to Text-Guided 3D Localization Without Ground Truth

Understanding GAN Mode Collapse: Causes and Solutions

Data Scarcity Solution: S-CycleGAN for CT-to-Ultrasound Translation

Optimizing SAGE Net: Achieving High Performance with Shorter Input Sequences for Online Inference

Model Promotion: Using EMA to Balance Learning and Forgetting in IIL

Evaluating Instance-Incremental Learning: CIL Methods on Cifar-100 and ImageNet

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps