Long-Tailed Continual Learning For
Visual Food Recognition

Food image recognition is a challenging yet practical task that is related to various real world applications such as the image-based dietary assessment [23, 24, 25]. The performance of nutritional content analysis such as energy [26] or macronutrients are heavily reliant on the accuracy of recognizing and predicting the correct food types from food images. Most existing deep learning based work leverage off-the-shelf models [27, 28, 11, 29] and train on static food image datasets [12, 13, 30, 31, 32, 9]. In order to address the issue of inter-class similarity and intra-class variability, a manually built hierarchy based food recognition is proposed in [10]. Later, Mao et al. [9] propose to construct a food hierarchy based on visual similarity and nutrition content [33] without requiring human efforts. In addition, food recognition has been studied under different scenarios such as ingredient recognition [34, 35], fine-grained recognition [36, 37, 38], few-shot learning [39, 40], long-tailed recognition [14, 41] and continual learning [42, 43]. However, none of the existing method are capable of learning new classes continuously in long-tailed distributions, which closely relates to real-world food recognition as new foods appear sequentially overtime with a minority of foods consumed more frequently than the others [14]. Though the most recent work [17] aims to integrate continual learning and long-tailed recognition, they decouple the training process to multi-stages and do not focus on food images. In this work, we target on long-tailed continual learning for visual food recognition to fill this gap and introduce a novel end-to-end framework to address both class-imbalance issue and catastrophic forgetting simultaneously.

Image recognition in long-tailed distribution has been widely studied in recent years [44]. Existing work can be mainly categorized into two main groups including re-weighting and re-sampling based methods. The major challenge of long-tailed recognition is the imbalance training data between instance-rich (head) and instance-rare (tail) classes. The re-weighting based methods aim to balance the loss or gradients during the training process. The class-level re-weighting loss assigns weights to each class such as the Balanced Softmax loss [45], Label-Distribution-Aware Margin loss [46] where the inverse of class frequency [47] is widely used as the class weights. In addition, re-sampling based techniques aim to construct a balanced training set by over-sampling the tail classes or under-sampling the head classes. However, the naive over-sampling [48] by simply repeating the tail classes data for balanced training further intensifies the over-fitting issue and the naive under-sampling [49] by randomly removing the training data of head classes results in performance degradation. Therefore, most existing work performs efficient data augmentation techniques to improve the generalization ability of tail classes and achieve better overall recognition performance. Gao et al. [41] propose Dynamic Mixup for multi-label long-tailed recognition problem, which dynamically adjusts the selection of images based on the previous training performance and set the label of synthetic image as the union of two images. The most recent work proposed CMO [21], which applies CutMix [22] for data augmentation by cutting the foreground region in tail classes images and paste in head classes background. The center idea of CMO is to leverage the context rich information from the head classes to help the generalization of tail classes. Later, He et al. [14] improves the CMO to use visually similar image pairs and allows for multi-image CutMix to achieve improved performance. In spite of the efficiency of CutMix for data augmentation, one of the limitations is that it suffers from loss of semantic information of the original image since the cut region is generated randomly. Inspired by [50], we introduce a novel CAM based CutMix, which is able to seamlessly combine the images without losing semantic information, which will be illustrated in Section IV.

Continual learning, also known as incremental learning or lifelong learning, has been studied under different scenarios such as class-incremental, task-incremental and domain-incremental learning as categorized in [51]. From the perspective of real world application, we primarily focus on class-incremental learning in this work, which aims to learn new classes continuously and perform classification on all classes seen so far during the inference phase, which does not require the task index or use multi-head classifier [52] compared to task-incremental learning. The domain-incremental learning instead does not involve new classes but only the domain shift for learned knowledge. The major challenge of class-incremental learning is catastrophic forgetting [15] where the updated model quickly forgets the learned knowledge due to the unavailability of learned classes data. There are two main groups of existing class-incremental learning methods including regularization based and replay based methods.

Regularization based methods address forgetting by restricting the change of learned parameters during the learning of new classes. The initial work freeze the learned parameters in fully connected layer [53] or constraining the weights update [54], which also confines the model’s ability to learn from new data. Later, Li et al. [55] proposed to use knowledge distillation [18] to maintain the learned knowledge by mimicking the output logits distribution from a teacher model, which is then improved in [56] by adding stronger constraints. In addition, feature-based knowledge distillation for class-incremental learning are applied in [57] by minimizing the discrepancy of learned representations between student and teacher models, which is further developed in [58] integrating the logits and feature-based distillation. However, existing knowledge distillation methods are developed on balanced data and becomes less efficient in long-tailed distribution. As revealed in [19], the knowledge distillation using output logits may even harm the overall performance if the teacher model is not trained on balanced data due to the bias towards instance-rich classes. Furthermore, directly applying feature based distillation may also impose new challenge such as feature space misalignment [20] due to the evolving of feature space when learning new classes especially in a long-tailed scenario where the data distribution may vary a lot for each incremental learning step. We address this problem by adding a prediction head to map the current representation space to the past, enabling more efficient knowledge transfer. Replay based methods assume the availability of a memory budget to store a small of number of learned classes data as exemplars to perform knowledge replay during the class-incremental learning. Herding algorithm [59] for exemplar selection based on the class mean vector is firstly proposed in [60], which becomes one of the most popular strategies in existing work [56, 61, 62, 57, 63]. Besides, He et al. [42] further improves herding by applying an additional clustering step to select exemplars from each cluster based on herding, which mitigates the issue of intra-class variability. Despite the effectiveness of exemplar replay to maintain the learned knowledge, existing methods assume the training data is balanced and each class has more training data than the memory budget (e.g. 20 exemplars per class). However, the majority of classes in the long-tailed scenario only contain a few training samples, resulting in class-imbalance issues in the exemplar set and may even harm the overall performance when performing the knowledge replay. In this work, we address this issue by constructing a balanced exemplar set by augmenting the tail classes data with the proposed CAM based CutMix, which not only improves the efficiency of knowledge replay to maintain the knowledge but increases the generalization ability on tail classes to achieve better overall performance.

We focus on continual learning in class-incremental settings where the objective is to learn new classes incrementally and perform classification on all classes seen so far during the inference phase. Specifically, the continual learning in the class-incremental scenario can be formulated as applying an initial model $h_{0}$ to learn a sequence of $N$ tasks denoted as $\mathcal{T}=\{\mathcal{T}^{1},...,\mathcal{T}^{N}\}$ where each task $\mathcal{T}^{i}$ contains $C_{i}$ non-overlapped new classes, which is also known as the incremental step size. During the learning phase of each new task, only the training data $D_{i}=\{\textbf{x}_{i}^{j},y_{i}^{j}\}$ of the current task is available where $\textbf{x}_{i}^{j}$ and $y_{i}^{j}$ denote the $j$-th input image and label, respectively. After each incremental learning step, the updated model $h_{i}$ needs to classify $C_{1:i}$ classes encountered so far. The major challenge of continual learning is catastrophic forgetting [15] where the updated model $h_{i}$ after learning the task $\mathcal{T}^{i}$ forgets the knowledge of previous tasks $\{\mathcal{T}^{1},...,\mathcal{T}^{i-1}\}$, resulting in significant performance degradation to classify $C_{1:i-1}$. In the conventional setup, the training data $D_{i}$ for each task $\mathcal{T}^{i}$ is balanced distributed, containing $|D_{i}|/C_{i}$ samples per class. However, this assumption simplifies the real world complexities especially for food recognition where data is usually long-tailed distributed and exhibits imbalance among food classes. Formally, the training data $D_{i}$ for each task in long-tailed continual learning is a class-imbalanced distributed with each class containing $(0,|D_{i}|)$ training samples. The entire training data $D$ for all the $N$ tasks $\mathcal{T}$ exhibits the long-tail distribution.

Despite the effectiveness of knowledge distillation in conventional continual learning setup as described in Section IV-A1, it is difficult to apply when new classes exhibit long-tailed distribution since the output logits of the teacher model could be heavily biased towards instance-rich classes due to the severe class-imbalance problem [62, 63]. Directly applying knowledge distillation as in (2) on biased output logits may even harm the overall performance [19]. Therefore, we explore the feature-based knowledge distillation in this work for more efficient knowledge transfer in the long-tailed continual learning scenario. As discussed in [20], one of the challenges when applying feature-based distillation is called the feature space misalignment where the representation of student and teacher models could mismatch in terms of both magnitude and direction. This problem is also relevant in the long-tailed continual learning scenario as the new knowledge is acquired sequentially, which requires the evolution of the continual learning model in the feature space to accommodate the addition of new classes. In this work, we address this challenge by introducing a simple yet effective method as shown in Figure 4. Specifically, instead of directly mimicking the feature from teacher model, we apply an additional predictor $g$ on the head of the continual learning model to map the current representation space to the past in the teacher model. Given an image x, the predictor $g$ takes the feature representation from the current model (i.e. student model) $h_{i}(\textbf{x})$ as input and outputs the mapped feature $g(h_{i}(\textbf{x}))$. Then we distill the knowledge from the teacher model $h_{i-1}$ by

where $<,>$ measures the cosine similarity. By applying the predictor, we provide the student model with more freedom to accommodate the previous learned representation into the current feature space, enabling more efficient knowledge distillation in long-tailed continual learning. The predictor $g$ is be removed after each incremental learning phase. Note that although we apply cosine embedding loss for knowledge distillation, our method can be integrated with other loss functions such as the Mean Squared Error (MSE) loss.

Existing exemplar replay based methods assume each class should contain at least $\mathcal{M}$ images given $\mathcal{M}$ as the memory budget in Section IV-A2. However, most classes in long-tailed distribution may contain only a few training samples $n<\mathcal{M}$, which imposes two new challenges including (i) inefficiency of knowledge replay due to the insufficient of training samples and (ii) intensification of the class-imbalance issue if we directly combine the stored exemplars with training data from new class due to the imbalanced nature of memory buffer. Therefore, we propose a novel data augmentation method in this work to construct a balanced exemplar set by augmenting the tail classes images to address both aforementioned issues. The overview of proposed data augmentation technique is illustrated in Figure 5. To address the issue of losing semantic information when performing data augmentation [21, 14] as described in Section II-B, we propose to use a class activation map (CAM) [68] to identify the most important region from instance-rare classes images and then preserve the semantic information by performing CutMix [22] to cut and paste the identified region into the images with rich context that are selected based on visual similarity. Specifically, we construct class-balanced memory buffer before each new task $\mathcal{T}^{i+1}$ by augmenting stored images for food classes $C_{t}$ with less than $\mathcal{M}$ exemplars through CutMix in conjunction with images selected from food classes $C_{h}$ containing $\mathcal{M}$ exemplars. Given an input image $\textbf{x}_{t}\in C_{t}$, we first select the most visually similar candidate $\textbf{x}_{h}\in C_{h}$ by comparing the cosine similarity with $h_{i}$ as feature extractor where $\textbf{x}_{h}=\underset{\textbf{x}_{k}\in C_{h}}{\text{argmax}}<h_{i}(\textbf{x}_{t}),h_{i}(\textbf{x}_{k})>$. The lower half of Figure 5 illustrates the procedure to identify the the region to cut and paste into $\textbf{x}_{h}$. Formally, given $\textbf{x}_{t}\in\mathbb{R}^{c\times h\times w}$, the class-activation map $M(\textbf{x}_{t})\in\mathbb{R}^{h\times w}$ is calculated by

where $v_{y_{\textbf{x}_{t}}}\in\mathbb{R}^{d}$ refers to the weight vector in classifier of the current model corresponding to the seen class $y_{\textbf{x}_{t}}\in C_{1:i}$. The value of CAM ranges from $[0,1]$ and a higher value indicates the more discriminative class-specific region. Therefore, we apply a random threshold $\sigma\in(0,1)$ to select the region $M(\textbf{x}_{t})^{T}\in\mathbb{R}^{h\times w}$ where

without losing semantic information of the input image. Finally, we perform CutMix to generate a synthetic image $\tilde{\textbf{x}}_{t}$ by

where $\odot$ refers to element-wise multiplication and $S(\textbf{x}_{t})^{T}$ denotes the binary mask obtained from $M(\textbf{x}_{t})^{T}$ that 1 indicates the region with $M(\textbf{x}_{t})^{T}>0$. The class label $\tilde{y}_{t}$ of the synthetic image is calculated by the area of the replaced region in $\textbf{x}_{h}$ as

where $A_{r}$ and $A$ denote the area of the replaced region and the total area of $\textbf{x}_{h}$. $y_{h}$ and $y_{t}$ refer to the original class label for $\textbf{x}_{h}$ and $\textbf{x}_{t}$, respectively. Note that the exemplar augmentation is performed at the beginning of each new task and the augmented images are not stored in the memory buffer. Note that the Grad-CAM[69], which can be regarded as the generalization of CAM [68], could also be applied in our method.

While the exemplar augmentation mitigates the class-imbalance issue by constructing a balanced memory buffer, the number of available training data between new classes and the stored classes may still vary a lot during the training phase due to the limited memory budget. Existing work [17] addresses this problem by decoupling the training process into two stages to first learn a feature extractor and then fine-tune the classifier using class-balanced sampler. In this work, we propose to use Balanced Softmax (BS) [45] by extending it into a long-tailed continual learning scenario without requiring a decoupled training process. Specifically, during the training phase of the new task $\mathcal{T}^{i}$, a distribution vector $v_{d}\in\mathbb{R}^{C_{1:i}}$ is generated by counting the number of training data of each food class for input images in the current task. Recall $L\in\mathbb{R}^{C_{1:i}}$ is the output logits from current model $h_{i}$, the distribution vector is then used as the prior information when calculating the loss as shown in (10)

where $\bar{v}_{d}=v_{d}/sum(v_{d})$ is the normalized distribution vector and $\Phi()$ denotes the $Softmax$ function. Therefore, the larger value in the distribution vector achieves smaller gradients when we compute the cross-entropy using the adjusted logits $v_{t}+L$ and vice versa, which address the class-imbalance issue and enables the end-to-end training pipeline.

The overall training loss function is the weighted sum of feature-based knowledge distillation as described in (5) and the balanced softmax $\mathcal{L}_{bs}$, which can be expressed as

where $\lambda$ is the adaptive ratio to tune the two losses. In this work, as the number of training data $D_{i}$ may vary a lot for each task $\mathcal{T}^{i}$, we propose to calculate $\lambda=\sqrt{|D_{i}|/|D_{1:i}|}$ as the ratio of training data for the current task and the learned tasks. Therefore, the ratio $\lambda$ increases when there are more training data from new classes.

We conduct experiments on four benchmark long-tailed food image datasets including: (1) Food101-LT [14], (2) VFN-LT [14], (3) VFN-INSULIN and (4) VFN-T2D where the latter two datasets are introduced in Section III.

Food101-LT is the long-tailed version of Food-101 [12] constructed by applying Pareto distribution [71] with the power ratio $\alpha=6$. We randomly partition the 101 food classes into 5, 10, and 20 tasks for continual learning. Therefore, each task comprises 20, 10, and 5 new classes, respectively, with the exception of the first task, which includes one additional class. The test set is kept as balanced with 125 images per class.

VFN-LT, VFN-INSULIN and VFN-T2D are all the long-tailed version of VFN [9] but constructed based on the food consumption frequency of different populations including (i) healthy people (ii) insulin takers and (iii) individuals with type 2 diabetes without taking insulin, respectively. We randomly divide the 74 food classes into $N=7$ tasks with the first task contain 14 new classes and the remaining tasks contain 10 new classes for continual learning. The test set is balanced with 25 images per class. To facilitate an equatable analysis, we use the same testing data in VFN-LT, VFN-INSULIN and VFN-T2D, which is balanced as 25 samples per class with total 1,850 images.

Our implementation of neural networks are based on Pytorch framework [72] and we apply the ResNet-18 from scratch as the backbone network for experiments for all datasets. The ResNet implementation follows the setting suggested in [11]. We train 90 epochs for each new task with the learning rate starts from 0.1 and decreased with ratio $1/10$ for every 30 epochs. The batch size is set as 128 and we apply the stochastic gradient descent (SGD) optimizer with weight decay $0.0001$. The memory budget is set as $\mathcal{M}=20$ to store at most 20 images per food class in the memory buffer.

Evaluation protocol: We use Top-1 classification accuracy as the evaluation metric and evaluate the updated model after learning each new task $\mathcal{T}^{i}$ on test data belonging to all classes seen so far $C_{1:i}$. Besides, we report the last step accuracy $A_{L}$ defined as the classification accuracy on the entire test set after learning all $N$ tasks and the average accuracy $A_{M}$ calculated by averaging the accuracy obtained after learning each new task, which shows the overall performance for the entire continual learning procedure. Each experiment is run five times and the average performance is presented.

Existing methods: We compare our proposed framework with existing continual learning methods including LwF [55], EWC [54], iCaRL [60], EEIL [61], LwM [70], IL2M [19], BiC [62], LUCIR [57] and PODNet [58]. Also, we apply the 2-stage framework for long-tailed continual learning in [17] and incorporate it into three existing methods including EEIL-2stage, LUCIR-2stage and PODNet-2stage for comparisons.

Table II summarizes the results on Food101-LT, VFN-LT, VFN-INSULIN and VFN-T2D in terms of the last step accuracy $A_{L}$ and the average accuracy $A_{M}$. We observe noticeable improvements for both $A_{L}$ and $A_{M}$ of our proposed method by comparing with existing work especially on VFN-LT, VFN-INSULIN and VFN-T2D, which contains a heavier tail with a larger imbalance ratio as introduced in Section III. For example, we achieve around $5\%$ increase of average accuracy $A_{M}$ and $2\%$ increase of the last step accuracy $A_{L}$ on VFN-INSULIN compared with the 2-stage framework even without requiring the decoupled training process. Furthermore, we obtain the best performance on Food101-LT with different number of tasks $N\in\{5,10,20\}$. We notice both $A_{L}$ and $A_{M}$ accuracy will drop as the total number of tasks $N$ increase since we need to address the catastrophic forgetting to maintain the learned knowledge at each learning phase of new tasks except for the fist task.

Figure 6 shows the results of top-1 classification accuracy evaluated on all the classes seen so far after learning each new task. Our method achieve promising performance at each learning phase of new task. In addition, different from conventional continual learning where the accuracy usually drops monotonously overtime when new tasks arrive due to the forgetting of previous knowledge, we notice that the performance may even increase after learning a new task in the long-tailed scenario. For example on Food101-LT with $N=5$, we observe the increase of accuracy for most methods after learning the third task. This occurs because the number of training samples varies significantly among different tasks in long-tailed continual learning where the model gains better knowledge for tasks with a larger number of training images. However, this imposes new challenges of learning from class-imbalanced data for each new task and also hyper-parameter tuning (e.g. the factor of knowledge distillation term as in Equation 4). In this work, we not only address the catastrophic forgetting by introducing a feature-based knowledge distillation and an effective exemplar augmentation technique but also alleviate the class-imbalance by integrating the balanced softmax with an adaptive ratio to adjust the impact of each loss term as illustrated in Equation 11 and strike a balance of stability-plasticity.

In this section, we evaluate the effectiveness of each individual component in our proposed framework including (i) the feature-based knowledge distillation ($\bm{\mathcal{L}_{fkd}}$), (ii) the cam-based data augmentation (CAM-CutMix) and (iii) the integration of balanced softmax with adaptive ratio ($\bm{\mathcal{L}_{bs}}$). Formally, we consider the baseline method as using imbalanced memory buffer ($\mathcal{M}=20$) with cross-entropy loss and integrating each of the aforementioned components to conduct experiments. The results in terms of average accuracy $A_{M}$ are summarized in Table III. We observe consistent performance improvements compared with baseline by adding our proposed techniques. Specifically, the feature-based knowledge distillation $\bm{\mathcal{L}_{fkd}}$ achieves the largest improvements on Food101-LT dataset, demonstrating that catastrophic forgetting is a crucial issue and the integration with CAM-CutMix is able to achieve higher accuracy. On the other hand, as VFN-LT, VFN-INSULIN, and VFN-T2D exhibit more severe class-imbalance issue due to higher imbalance ratio, the balanced softmax $\bm{\mathcal{L}_{bs}}$ term has the most significant impact, resulting in the largest performance improvements. Our proposed framework by integrating all the three components obtains the best classification accuracy on all datasets.

We also evaluate our proposed CAM-CutMix by replacing it with existing data augmentation based methods including the CutMix [22] based approaches: (a) the original CutMix used in CMO [21], (b) Visual-Multi CutMix (VM-CMO) [14], (c) SnapMix [50] and the Mixup [73] based approach: (d) D-Mixup [41]. We conduct experiments on VFN-LT and Food101-LT with $N=10$ as shown in Table IV. Generally, the CutMix based approaches work better in long-tailed continual learning scenarios than D-Mixup, which is usually applied in multi-label recognition scenarios. In addition, the SnapMix achieve a slightly better performance than CMO and VM-CMO as it also considers the class-activation map (CAM) when generating mixed labels. Our method achieves the best performance as it not only preserves the most important regions based on CAM but also enables seamless CutMix, rather than relying on a randomly generated bounding box. The example augmented food images using VFN-LT are shown in Figure 7. Note that we do not visualize SnapMix [50] as it has the same synthetic image as in CMO [21] but with a different mixed label.

Despite the performance improvements our proposed framework demonstrates in comparison to existing methods as shown in Table II, the deployment in real-world applications still remains challenging based on the current classification accuracy. Therefore, in this section we discuss potential techniques that could be applied to boost the performance including (1) increaseing the memory buffer capacity to store more exemplar images for knowledge replay and (2) performing transfer learning by pre-training the backbone on large scale image datasets.