Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Task

The methods of this class usually assume that a common feature representation can be learned from individual tasks. According to the implementation manners, they likewise can roughly be divided into three sub-types:

1) Selective sharing of task features: for the tasks in the same subspace, they realize sharing by specifically regularizing the features among tasks. Typically, [35] uses the $\ell_{2}$ norm to regularize the task weight matrices to extract shared features for the test score prediction of most school students. [36] uses the $\ell_{1,2}$ norm to regularize the weight matrices to extract shared features between tasks for learning multi-tasks with different feature dimensions. [37] uses $\ell_{2,1}$ norm to regularize the weight matrices of various modal tasks to jointly select common features for multi-modal classification of Alzheimer’s disease.

2) Priori knowledge sharing of tasks: for the tasks defined in the same subspace, they use the same prior knowledge among tasks to realize sharing. Typically, [38] embeds prior knowledge (i.e., pathological images with different magnification belong to the same subclass) into the feature extraction process among different tasks to verify the relationship between tasks and pathological image categories for fine-grained classification and pathophysiological image classification. [39] uses a kind of meta data (i.e., contextual attributes) as a priori knowledge to capture the relationship between different tasks for multiple tasks clustering. [40] uses the same subclass of the gland area as the prior information in the convolutional neural network to guide the network inference for pathological colon image analysis.

3) Transformation sharing of task features: for the tasks represented in the same subspace, they realize sharing by performing the nonlinear transformations of the original feature representation among tasks. Typically, [41] uses a set of non-linearly transformed feature sharing units for image semantic segmentation and normal estimation. [42] uses the feature adapter to learn the non-linear transformation of the tasks features to automatically evaluate the child’s speech ability.

The methods of this class usually associate different tasks with their partial model parameters or weights to realize sharing. According to their learning manners used, they can roughly be divided into three sub-types:

1) Weighted sharing of weight matrices: for the tasks represented in the same subspace, they realize sharing by weightedly combining a set of weight matrices among tasks. Typically, [43] weights the weight matrices among tasks for boundary classification of keywords. [44] partitions the weight matrices among tasks into common and private parts, then weights the common part for multi-label classification. [45] weights the weight matrices at the same spatial position in the pictures and transfers them to each task for image depth estimation, segmentation, and surface normal prediction.

2) Common factor sharing via decomposing individual weight matrices: for the weight matrix of each task model, they decompose these matrices into private and common parts, where the common part is used for sharing. Typically, [46] decomposes the weight matrices of multiple task models into common and private parts, and further uses the common part for visual target tracking. [47] sparsely decomposes the parameter tensor of the prediction model into multiple parameter matrices, and linearly combines the corresponding parameter matrices into a set of base matrices for sharing. [48] decomposes a collective matrix of drug-disease correlations to share the correlation matrix between them for drug discovery.

3) Low-rank structure sharing of model weight matrices: for the tasks represented in the same subspace, they capture the low-rank structure of the weight matrix among tasks by specifically regularizing to realize sharing. Typically, [49] uses feature tensor flattening of different tasks (i.e., a convex combination of matrix trace norms) to capture its low-rank structure for multi-task learning. [50] uses a set of low-rank matrices to capture the potential relationships between multiple tasks for Parkinson’s disease diagnosis. [51] uses a set of low-rank matrices constrained by the nuclear norm for target detection in hyper-spectral images.

Given a big auxiliary task $T_{aux}$ and a dataset $\mathcal{D}_{aux}=\left\{x_{i},y_{i}\right\}_{i=1}^{N}$ containing $N$ samples with $x_{i}\in\mathbb{R}^{d}$ and its associated label $y_{i}\in\left\{1,\ldots,c\right\}$, where ${d}$ and ${c}$ are the numbers of dimensions and classes in the dataset $\mathcal{D}_{aux}$, respectively. Meanwhile we are given $M$ individual tasks $\left\{{T_{j}}\right\}_{j=1}^{M}$, and corresponding training dataset $\mathcal{D}_{j}=\left\{x_{k}^{j},y_{k}^{j}\right\}_{k=1}^{N_{j}}$ with ${N_{j}}$ samples, $x_{k}^{j}\in\mathbb{R}^{d}$ and its associated label $y_{k}^{j}\in\left\{1,\ldots,c_{j}\right\}$, where ${c_{j}}$ is the number of classes in the dataset $\mathcal{D}_{j}$. We assume that the classset $\mathcal{C}_{T_{aux}}$ of the auxiliary task contains all the individual tasks classes $\mathcal{C}_{T}$, namely, $\mathcal{C}_{T_{aux}}=\mathcal{C}_{T_{1}}\cup\mathcal{C}_{T_{2}}\cup,...,\mathcal{C}_{T_{M}}$, where $C_{T_{i}}$ and $C_{T_{j}}(i\neq j)$ can be partially overlapped, or even unoverlapped. This makes DAMTL applicable under more general settings than most existing MTL methods.

To learn the above tasks mentioned in Section 3.1, we first assume that this big auxiliary task network has $L$ convolutional layers and $H$ fully connected (FC) layers, and pre-train the network to obtain two sets of weights $\mathrm{W}_{T_{aux}}=\left\{\mathrm{W}_{aux}^{1},\mathrm{W}_{aux}^{2},\ldots,\mathrm{W}_{aux}^{L}\right\}$ and $f_{T_{\text{aux }}}=\left\{f_{\text{aux }}^{1},f_{\text{aux }}^{2},\ldots,f_{\text{aux }}^{H}\right\}$, where ${\rm{W}}_{aux}^{\emph{l}}$ and $f_{\text{aux }}^{h}$ represent the weights of the $l$-th convolutional layer and the weights of the $h$-th FC layer, respectively. Then, we manage to acquire necessary knowledge selectively from the trained big task to learn individual tasks by soft masking the weights of each corresponding convolutional layer, as shown in Fig 3. In what follows, we take the task $T_{j}$ as a training example. To selectively extract specific knowledge layer-wisely from the big auxiliary task network, we introduce a soft masking matrix $\Psi_{T_{j}}^{l}$ as follows:

here $S_{T_{j}}^{l}$ denotes the extracted knowledge from the auxiliary task, which will be transferred to the $l$-th convolutional layer in task $T_{j}$ , $\Psi_{T_{j}}^{l}$ just takes non-negative value, and $\odot$ denotes the Hardmard product. Then, we use Eq.(2) to activate the multiplication of $S_{T_{j}}^{l}$ and $F_{T_{j}}^{l-1}$ to realize the knowledge being transferred

where $F_{T_{j}}^{l}$ is the feature representation of the $l$-th convolutional layer of the task $T_{j}$ network, $\sigma$ is the activation function, and $\mathrm{b}_{T_{j}}^{l}$ is the bias vector.

Now optimizing the $S_{T_{j}}^{l}$ boils down to optimizing $\Psi_{T_{j}}^{l}$. Specifically, in order to ensure the prediction accuracy of the DAMTL network, we leverage Conditional Maximum Mean Discrepancy (CMMD) [52] to align the conditional probability distributions between $T_{aux}$ and $T_{j}$ in the FC layers to alleviate the possible class drift in the knowledge transfer from $T_{aux}$ to the $T_{j}$, which is expressed as:

where $(O_{k}^{h-1})$ is the representation of the task $T_{j}$ in the $(h-1)$-th FC layer, $f_{\emph{aux}}^{h}$ and $f_{T_{j}}^{h}$ are the weights in the $h$-th FC layer, and $k_{c_{j}}^{T_{j}}$ is the number of samples of the $c_{j}$-th class in task $T_{j}$.

In practice, we use the inputs $x_{k}$ (${k}=1,2,\ldots,n$) and the labels $y_{k}$ (${k}=1,2,\ldots,n$) to minimize the following individual task loss

where $\hat{y}_{k}^{T_{j}}$ is the predicted output, $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters, the first term is the cross-entropy loss of task $T_{j}$, and the second term uses the $\ell_{1}$ norm to make the soft masking matrix $\Psi_{T_{j}}^{l}$ spare. Finally, we conduct joint training of these tasks with the total loss

where $\alpha_{T_{j}}$ is the hyper-parameter to be adjusted.

The whole process of the proposed method to solve DAMTL is summarized in Algorithm (1).

We conduct experiments on the following data sets and divide $70\%$ of the data as training set and the remaining $30\%$ as testing set. The detailed information is shown in Table 2 and Table 3.

The ImageNet Dataset¹¹1http://image-net.org/download is a computer vision dataset, which is used as an auxiliary task in the experiment. The dataset contains 21,841 categories and 14,197,122 images.

The Office-Caltech Dataset²²2https://people.eecs.berkeley.edu/~jhoffman/domainadapt/ is divided into two datasets, Office-Caltech10 and Office-Caltech31, and each dataset has 2,533 images, which are composed of three different subsets Dslr, Amazon and Webcam. We randomly select 10 categories in each subset to do experiment.

The Office Home Dataset³³3http://hemanthdv.org/OfficeHome-Dataset/ is composed of Art, Clipart, Product, Real-World, and each subset covers 15500 images from 65 categories. We also randomly select 10 categories in each subset as the dataset.

The Caltech-256 Dataset⁴⁴4http://www.vision.caltech.edu/Image_Datasets/Caltech256/ is a dataset collected by the California Institute of Technology. It has 256 categories, and each category has more than 80 images and a total of 30,607 images. We also randomly select 10 categories in this data set for experiment.

The Tiny ImageNet Dataset⁵⁵5https://www.kaggle.com/c/tiny-imagenet is a dataset collected by Stanford University, which is a subset of the ImageNet dataset. The dataset contains 200 categories and a total of 100,000 images, and the size of each image is $3*64*64$. We randomly select 10 categories from the Tiny ImageNet Dataset for experiments.

For evaluation, we use common single-task and multi-tasks network architectures to train each task separately/jointly, and its experimental results serve as our single-task and multi-tasks baseline. Simultaneously, we compare our proposed method with the following MTL methods:

Single task⁶⁶6https://github.com/machrisaa/tensorflow-vgg [53]: This method uses a single VGG network to learn the predictive model for each independent task.

Multi-task⁷⁷7https://github.com/luntai/VGG16_Keras_TensorFlow: This method uses multiple identical VGG networks to jointly learn a multi-task prediction model.

Cross-Stitch⁸⁸8https://github.com/helloyide/Cross-stitch-Networks-for-Multi-task-Learning [54]: This method uses a cross stitch unit to learn the common features in two network feature layers for learning multiple tasks.

NDDR-CNN⁹⁹9https://github.com/ethanygao/NDDR-CNN [55]: This method uses the NDDR module to automatically integrate the features of each sub-network layer for MTL.

MTAL¹⁰¹⁰10http://parnec.nuaa.edu.cn/3021/list.htm[12]: This method leverages the similarity between convolution kernels to capture common knowledge among multi-network for joint learning of various tasks.

DAMTL¹¹¹¹11http://parnec.nuaa.edu.cn/3021/list.htm: This method uses the abound knowledge of a lager auxiliary task to help joint learning of multiple tasks.

In the contrasted deep neural network methods, we adjust the hidden units, learning rate, and the number of training steps in each layer according to the parameter settings of the corresponding reference. In DAMTL, we adjust the hyper-parameters in the same way. Specifically, for all experiments, we set the learning rate $\eta$ to 0.01, $\alpha_{T_{j}}$ to 0.01, $\lambda_{1}$ and $\lambda_{2}$ are 0.9. In addition, in DAMTL network training, we select the rectified linear unit (ReLU) function as the activation function $\sigma$. All the deep learning models are implemented by Tensorflow.

We conduct experiments on the above-mentioned datasets and compare our methods with the state-of-the-arts (SOTAs), while analyzing the experimental results.

First, the performance of the DAMTL method shown in Table 4 and Table 5 is significantly better than other methods. In the PO-2 group of experiments, the DAMTL method is better than the MTAL methods.

Secondly, Table 4 shows that most of the MTL methods are better than the single-task learning method, which demonstrate that using the relationship between tasks to capture their useful information for interaction can promote the effectiveness of the joint learning of multiple tasks. In addition, we find that different multi-task learning methods have different performance results, which are caused by the differences among tasks. For example, the experimental results of the dataset Caltech-101 in PO-1 show that the Cross-Stich method and the NDDR-CNN method are lower than the accuracy of the single-task learning method.

Finally, the performance of DAMTL is better than that other methods in all datasets, which shows that extracting and transferring features from a big auxiliary task (which contains the class label information in all individual tasks) can help the joint learning of multiple tasks with partially overlap or even unoverlapping label sets and further improve DAMTL performance.

Also, Fig.4 shows the performance comparison of the mean and standard deviation of the classification accuracy of various methods in the two categorical experiments. We observe that the overall performance of the DAMTL method is better than that other methods. The above experimental results are consistent with our theoretical analysis.

The result is shown in Fig.5 (a), we observe that when the label sets among tasks partially overlap, although the single-task method takes the shortest time, it does not use shared information, so the accuracy is the lowest. The NDDR-CNN method takes the longest time, but the accuracy is the second lowest (even worse than the multi-task benchmark). This indicates that the differences among tasks lead to adversarial interference in the learning process of this method. In addition, the multitasking, Cross-Stich, NDDR-CNN and MTAL methods are comparable in time cost, and our DAMTL method has the highest accuracy. As shown in Fig.5 (b), we observe that when the label sets among tasks do not overlap, the single-task method takes the shortest time, but its accuracy is the lowest. MTAL uses the convolution kernel sharing technology, thus spending the second shortest time. In conclusion, our method has relatively low time cost and the highest accuracy in scenarios where the task label sets are partially overlapped or even unoverlapped.

In this section, we demonstrate that the addition of Eq.(3) to DAMTL through ablation analysis is very important to achieve state-of-the-art performance. We first study the performance of multiple tasks with overlapping partial label sets without using Eq.(3). The performance of DAMTL network on Caltech-101, Caltech-256, Amazon, Webcam, Dlsr, and Product is shown on Table 6. Obviously, when the rest of the data set except Product, the performance of DAMTL is reduced.

Then, we delete the Eq.(3) part for unoverlapping label sets between tasks. The performance of DAMTL network on Art, Real World, Caltech-101, Webcam, Amazon, and T-ImagNet is shown on Table 7. The performance of DAMT on all data sets is significantly degraded. Finally, we use Eq.(3) DAMTL network through ablation analysis which can effectively improve performance.

In this section, we analyze the convergence of our proposed method on two benchmarks, i.e., Caltech-101 and Caltech-256 and the values of the objective function (5) with respect to iterations on the two datasets are shown in Fig.6 (a) and (b) respectively. From Fig.6 (a), we can see that the loss curves on the data sets Caltech-101 and Caltech-256 tend to converge after about 50 iterations. In Fig.6 (b), the loss curves on the data sets Art and Real-world stabilize after about 70 and 50 iterations, respectively.