Class-Incremental Lifelong Learning in Multi-Label Classification

In this study, each data is trained only once in the form of a data stream. Given $T$ recognition tasks with respect to train datasets $\{\mathcal{D}^{1}_{\text{trn}},\cdots,\mathcal{D}^{T}_{\text{trn}}\}$ and test datasets $\{\mathcal{D}^{1}_{\text{tst}},\cdots,\mathcal{D}^{T}_{\text{tst}}\}$. For the $t$-th task, we have new and task-specific classes to be trained, namely $\mathcal{C}^{t}$. The goal is to build a multi-label classifier to discriminate an increasing number of classes. We denote $\mathcal{C}_{\text{seen}}^{t}=\bigcup_{n=1}^{t}\mathcal{C}^{n}$ as seen classes at task $t$, where $\mathcal{C}_{\text{seen}}^{t}$ contains old class set $\mathcal{C}_{\text{seen}}^{t-1}$ and new class set $\mathcal{C}^{t}$, that is, $\mathcal{C}_{\text{seen}}^{t}=\mathcal{C}_{\text{seen}}^{t-1}\cup\mathcal{C}^{t}$. Note that during the testing phase, the ground truth labels for LML classification contain all the old classes $\mathcal{C}_{\text{seen}}^{t}$.

Correlation Matrix is often built for multi-label learning (Chen et al., 2019, 2021), and can be used to construct label relationships. Our Augmented Correlation Matrix (ACM) provides the label relationships among all seen classes $\mathcal{C}^{t}_{\text{seen}}$ and is augmented to capture the intra- and inter-task label independences. Most existing multi-label learning algorithms (Chen et al., 2019, 2021) rely on constructing the inferring label correlation matrix $\mathbf{A}$ by the hard label statistics among the class set $\mathcal{C}$: $\mathbf{A}_{ij}=P(\mathcal{C}_{i}|\mathcal{C}_{j})|_{i\neq j}$. We construct ACM $\mathbf{A}^{t}$ for task $t>1$ in an online fashion to simulate the statistic value denoted as

in which we take four block matrices including $\mathbf{A}^{t-1}$ and $\mathbf{B}^{t}$, $\mathbf{R}^{t}$ and $\mathbf{Q}^{t}$ to represent intra- and inter-task label relationships between old and old classes, new and new classes, old and new classes as well as new and old classes respectively. For the first task, $\mathbf{A}^{1}=\mathbf{B}^{1}$. For $t>1$, $\mathbf{A}^{t}\in\mathbb{R}^{|\mathcal{C}_{\text{seen}}^{t}|\times|\mathcal{C}_{\text{seen}}^{t}|}$. It is worth noting that the block $\mathbf{A}^{t-1}$ (Old-Old) can be derived from the old task, so we will focus on how to compute the other three blocks in the ACM.

ACM is auto-updated dependencies among all seen classes. With the established ACM, we can leverage Graph Convolutional Network (GCN) to assist the prediction of CNN as Eq. (6). We propose an Augmented Graph Convolutional Network (AGCN) to manage the augmented fully-connected graph. AGCN is learned to map this label graph into a set of inter-dependent object classifiers. AGCN is a two-layer stacked graph model, which is similar to ML-GCN (Chen et al., 2019).Based on the ACM $\mathbf{A}^{t}$, AGCN can capture class-incremental dependencies in an online way. Let the graph node be initialized by the Glove embedding (Pennington et al., 2014) namely $\mathbf{H}^{t,0}\in\mathbb{R}^{|\mathcal{C}_{\text{seen}}^{t}|\times d}$ where $d$ represents the embedding dimensionality. The graph presentation $\mathbf{H}^{t}\in\mathbb{R}^{|\mathcal{C}_{\text{seen}}^{t}|\times D}$ in task $t$ is mapped by:

As shown in Fig. 2, together with an CNN feature extractor, the multiple labels for an image $\mathbf{x}$ will be predicted by

where $\mathbf{A}^{t}$ denotes the ACM and $\mathbf{H}^{t,0}$ is the initialized graph node. Prediction $\mathbf{\hat{y}}=[\mathbf{\hat{y}}_{\text{old}}~{}\mathbf{\hat{y}}_{\text{new}}]$, where $\mathbf{\hat{y}}_{\text{old}}\in\mathbb{R}^{|\mathcal{C}_{\text{seen}}^{t-1}|}$ for old classes and $\mathbf{\hat{y}}_{\text{new}}\in\mathbb{R}^{|\mathcal{C}^{t}|}$ for new classes. We train the current task for classifying using the Cross Entropy loss.

To mitigate the class-level catastrophic forgetting, inspired by the distillation-based lifelong learning method (Li & Hoiem, 2017; Zhou et al., 2022), we construct auto-updated expert networks consisting of CNN${}_{\text{xpt}}$ and AGCN${}_{\text{xpt}}$. The expert parameters are fixed after each task has been trained and auto-update along with new task learning. Based on the expert, we construct the distillation loss as

where ${\mathbf{\hat{z}}}=\sigma\left(\text{AGCN}_{\text{xpt}}(\mathbf{A}^{t-1},\mathbf{H}^{t-1,0})\otimes\text{CNN}_{\text{xpt}}\left(\mathbf{x}\right)\right)$ can be treated as the soft labels to represent the prediction on old classes. The $i$-th element $\hat{z}_{i}$ of ${\mathbf{\hat{z}}}$ represent the probability that the image $\mathbf{x}$ contains the class.

To mitigate the relationship-level forgetting, we constantly preserve the established relationships in the sequential tasks. The graph node embedding is irrelevant to the label co-occurrence and can be stored as a teacher to avoid the forgetting of label relationships. Suppose the learned embedding after task $t$ is stored as $\mathbf{G}^{t}=\text{AGCN}_{\text{xpt}}(\mathbf{A}^{t},\mathbf{H}^{t,0})$, $t>1$. We propose a relationship-preserving loss as a constraint to the class relationships:

By minimizing $\ell_{\text{gph}}$ with the partial constraint of old node embedding, the changes of AGCN parameters are limited. Thus, the forgetting of the established label relationships are alleviated with the progress of LML classification. The final loss for the model training is defined as

where $\ell_{\text{cls}}$ is the classification loss, $\ell_{\text{dst}}$ is used to mitigate the class-level forgetting and $\ell_{\text{gph}}$ is used to reduce the relationship-level forgetting. $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the loss weights for $\ell_{\text{cls}}$, $\ell_{\text{dst}}$ and $\ell_{\text{gph}}$. Extensive ablation studies are conducted for $\ell_{\text{gph}}$ after all relationships are built.

Multi-Task is the performance upper bound, and Fine-Tuning is the performance lower bound. In Tab. 1, our method shows better performance than the other state-of-art methods on the three metrics, as well as the forgetting value evaluated after task $T$. On Split-COCO, the AGCN outperforms the best of the state-of-art method PRS by a large margin (34.11% vs. 28.81%). The AGCN shows better performance than the others on Split-WIDE (41.12% vs. 37.93%), suggesting that AGCN is effective on the large-scale multi-label dataset. As shown in Fig. 3, which illustrates the mAP changes as tasks are being learned on two benchmarks. The proposed AGCN is better than other state-of-art methods through the whole LML process.

The final ACM visualization is shown in Fig. 4. The dependency between two classes with higher correlation has larger weights than irrelevant ones, which means the intra- and inter-task relationships can be well constructed even if the old classes are unavailable.

ACM effectiveness. In Tab. 2, if we do not build the relationships across old and new tasks, the performance of AGCN (Line 1) is already better than other non-AGCN methods, for example, 31.52% vs. 28.81% in mAP. This means only intra-task label relationships are effective for LML. When the inter-task block matrices $\mathbf{R}^{t}$ and $\mathbf{Q}^{t}$ are available, AGCN with both intra- and inter-task relationships (Line 2) can perform even better in all three metrics, which means the inter-task relationships can further enhance the multi-label recognition.