Model Composition: Can Multiple Neural Networks Be Combined into a Single Network Using Only Unlabeled Data?

Since the input model predictions are not always perfectly accurate, the generated pseudo-labels will be noisy, and therefore less reliable. These training examples could have an adverse impact on the training of the output model. We filter out such kind of examples, by employing an entropy-based thresholding mechanism.

For a given input $x$ and a network function $f$ such that $p=f(x)$, the entropy is given by $H(f,x)=-\sum_{i}p_{f_{i}}\log p_{f_{i}}$. An unreliable pseudo-label may be discarded if $H(f,x)>\tau$, for some threshold $\tau$. Although entropy thresholding does not guarantee a perfect filtering of bad pseudo-labels, but in practice it works well and has been used as a confidence indicator for similar purposes [Teerapittayanon et al.(2016)Teerapittayanon, McDanel, and Kung, Saporta et al.(2020)Saporta, Vu, Cord, and Pérez, Rottmann et al.(2018)Rottmann, Kahl, and Gottschalk]. Note that for some tasks such as object detection, models output a confidence score that can also be used for filtering bad pseudo-labels.

Next, in the pseudo-label aggregation phase, we employ a consensus based strategy, where the majority of the input models need to agree on a pseudo-label in order for it to qualify as a candidate and pass to the next step. Pseudo-label aggregation can be done in various ways such as unanimous (all models agree), affirmative (union of all predictions), consensus (majority voting), etc. [Casado-Garcıa and Heras(2020)]. Our experiments showed all these methods can be used with minor performance variations. We chose the consensus approach for the experiments since for combining a higher number of models, intuitively it makes more sense (See section 4 for a 10-model example). Note that for some tasks such as image classification, the aggregation will be a simple majority voting mechanism. For some other tasks such as object detection, it becomes more complicated due to the nature of the task. Here, we review our method of pseudo-label aggregation for object detection, which can also be extended to other similar tasks such as instance segmentation, tracking, etc.

Details of the pseudo-label aggregation strategy: Let $\cal\overline{D}$ denote the unlabeled dataset used. The input to the pseudo-label aggregation procedure is a list ${\cal S}=[\overline{\mathbf{y}}_{1},...,\overline{\mathbf{y}}_{N}]$, where each $\overline{\mathbf{y}}_{i}$ itself is a list of detections from an input model over all unlabeled training images in $\cal\overline{D}$. We then create a new list ${\cal S}_{im}=[p_{1},...,p_{|\cal\overline{D}|}]$ so that each $p_{i}$ contains predictions of all models on one single image, and length of ${\cal S}_{im}$ is equal to number of all images in $\cal\overline{D}$.

Next, for each element $p_{i}$, we unite the predictions by their category names and the overlapping of their bounding boxes. If the overlapped area of any two elements in $p_{i}$ is higher than a certain threshold, and meanwhile if these two elements are of the same category, then they are treated as detections of the same object, which are further grouped together into a sub-list: $\overline{p}_{i}=\{\overline{p}_{ij}\}$. Subsequently, we decide whether to keep each element $\overline{p}_{ij}$ depending on the number of unique models with predictions included in $\overline{p}_{ij}$, denoted by $K_{ij}$. In the most strict case, $\overline{p}_{ij}$ is kept in the list only when $K_{ij}=N_{ij}$, where $N_{ij}\leq N$ is the maximum number of models that may predict the object category corresponding to $\overline{p}_{ij}$; If we want a majority voting, then $\overline{p}_{ij}$ is kept when $K_{ij}\geq N_{ij}/2$; If a simple stacking strategy is used, then $\overline{p}_{ij}$ is kept regardless of $K_{ij}$. At this point, each $\overline{p}_{ij}$ could still have several candidate detections for the same region. Processing all of them through the detection network is not only cumbersome but could also decrease the overall performance. Therefore, we applied soft non-maximum suppression (Soft-NMS) [Bodla et al.(2017)Bodla, Singh, Chellappa, and Davis] to each $\overline{p}_{ij}$ to filter the predictions a second time. Algorithm 2 formally captures these steps and Figure 5 demonstrates an example.

Remark 1: $\overline{p}_{ij}$ for image $i$, represents a list of bounding boxes predicted on a particular object $j$, i.e. detections of a same object by different models. $K_{ij}$ is the number of unique models in $\overline{p}_{ij}$. $N_{ij}$ is the number of models that have the category of $\overline{p}_{ij}$ in their label set (i.e. number of models that actually have the capability of detecting that object category). As such, in general $K_{ij}\leq N_{ij}\leq N$. In an ideal case where all eligible models can detect an object $ij$, we will have $K_{ij}=N_{ij}$. If all input models have the same category label set, $N_{ij}=N$. In the case input models have different but overlapping categories (i.e. there is at least one category that is not supported by all models), for at least some $i$ and $j$, $N_{ij}<N$. If all models have strictly different categories (no overlap), $K_{ij}=N_{ij}=1$. And finally if some particular categories only belong to one model, for those categories $K_{ij}=N_{ij}=1$.

Remark 2: The experiments in Section 4 contain various practical scenarios in which different aspects of our method are evaluated. Moreover, Figure 9 in supplementary materials [Banitalebi-Dehkordi et al.(2021)Banitalebi-Dehkordi, Kang, and Zhang] shows a scenario where 10 input models with a diverse count and type of object categories are combined. For example in this figure, ‘teddy bear’ is only in $M_{1}$, ‘bicycle’ is in $M_{1},M_{3}$, ‘potted plant’ is in $M_{2},M_{3},M_{4}$, etc. In addition, we also explore in Section 4 the task of combining a face detection model with a mask detection one.

Once pseudo-labels are filtered and combined, they will be used to train the output model architecture. Any available labeled data will be used in a final fine-tuning stage to improve the performance. Note that pseudo-labels are generated from unlabeled data. This is because input model owners may only share an interface to their model API for inference, not necessarily the weights, code, architecture, training data, or labels, to protect their privacy. We treat the input models as black boxes. In other words, we only pass a set of arbitrary unlabeled images through them, and collect their predictions to use as pseudo-labels. This further allows us to choose an arbitrary architecture and size for the output model that combines the class categories of the input models. Consequently, our model composition method is agnostic to the training hyper-parameters of the input models such as various optimizers, learning rate schedules, batch sizes, etc.

It is also worth noting that this way of creating composite models can help with light domain shifts. As we see in section 4, input models trained on different datasets (for the same task) can still be effectively combined (even with different sets of categories). To what extent exactly our method can robustly support domain shifts remains out of the scope of this work, and we leave that as a future direction.

Selected model architectures: We have selected the task of object detection as the main experiments task due to its importance and wide-spread usage in practical applications. That being said, we will also provide results on the task of image classification, as it is often used as a baseline experiment task. For object detection, we utilized the following architectures: EfficientDet-D0 [Tan et al.(2020)Tan, Pang, and Le], EfficientDet-D1 [Tan et al.(2020)Tan, Pang, and Le], and RetinaNet-ResNet-50 [Lin et al.(2017)Lin, Goyal, Girshick, He, and Dollár]. For the classification task, we used: ResNet-18 [He et al.(2016)He, Zhang, Ren, and Sun], ResNet-152[He et al.(2016)He, Zhang, Ren, and Sun], and DenseNet-121[Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger].

Datasets: We used three sets of benchmarking datasets for object detection: COCO [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick], Pascal-VOC [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman], and Open-Images-V5 [Kuznetsova et al.(2018)Kuznetsova, Rom, Alldrin, Uijlings, Krasin, Pont-Tuset, Kamali, Popov, Malloci, Kolesnikov, et al.] (referred by OID hereafter). For classification, we use Caltech-256 [Griffin et al.(2007)Griffin, Holub, and Perona] and OID datasets.

Evaluation metrics: We follow the common practice by using the mean Average Precision, mAP @IoU=0.50:0.95, as the main metric to evaluate the performance of object detection models. We report top-1 accuracy for classification.

Training protocols and settings: We adopt code from [Eff()] for the object detection experiments, and use the same training hyper-parameters with ImageNet [Deng et al.(2009)Deng, Dong, et al.] pre-trained backbones. We trained all the models using SGD with a momentum of 0.9. We increased the learning rate from $8e^{-3}$ to $8e^{-2}$ for the first epoch and then trained the remaining 300 epochs using a cosine decay rule with a moving average decay set at 0.9998. Soft NMS was utilized to filter the pseudo-label detections in our method. We used an IoU threshold of 0.5 and a confidence threshold of 0.001. For the classification experiments, models were trained for 200 epochs, using an in-house code-base. An SGD optimizer with momentum 0.9 was used, and learning rate was exponentially increased from 0 to 0.01 for the first 8 epochs and then annealed down exponentially to 0.0001 in the remaining epochs.

Our experiments are categorized in various scenarios, which are explained in this subsection. These scenarios cover various possible cases of input models’ architectures, training data, and what kind of unlabeled data were used in our algorithm. Table 1 provides a summary of these scenarios. In Table 1, training data in each case is constructed from the training set of VOC, COCO, OID, or a subset of them (unlabeled). Validation sets are also built from the validation sets of VOC and COCO: a subset of COCO (union of input categories) for scenario 1, union of the val set of COCO & VOC for scenario 2 & 3, and val set of COCO for scenario 4. As such, validation sets may be different across different scenarios, but are the same within one scenario. Moreover, the class distributions of data for scenario 1 & 4 are shown in Table 2 and Figure 9 (supplementary), and for scenario 2 & 3 it follows the distributions of COCO & VOC. Next, we will go over the details of each experiment.

Scenario 1: Combining detectors with different expertise: We took 3 models, each trained on a subset of the COCO dataset but designed for a different purpose, one for detection of transportation related objects, one for sports related, and the other for home objects. These categories have some partial overlap. Table 2 shows the object categories used for each model. The combined model achieved by our model composition procedure combines the skills of the input models, and builds a stronger model with all object categories. We tried our method in two ways: one using unlabeled COCO images (similar data distribution to training data, but without using the labels), and the other using unlabeled OID (open images) dataset (entirely different dataset with different distribution). The upper-bound of the performance would be to train a model with all labels of all object categories (supervised). This model achieved 35.11% mAP on validation set of COCO (considering only object categories corresponding to the ones it was trained on). On the same validation set, our method achieved 32.61% when using unlabeled COCO, and 30.97% when using unlabeled OID. This shows that our method can effectively combine the models with different expertise, and achieve a performance close to that of the supervised upper-bound model. We further investigated the performance of our method if partial labels are available for fine-tuning, in a semi-supervised manner. Table 3 shows the results for this experiment. As observed, with fine-tuning, our method could even surpass the supervised model with 100% of labels.

Scenario 2: Combining input models that are trained on entirely different datasets. In scenario 1, input models had different expertise, by getting trained on different subsets of COCO (examples roughly came from a similar distribution). Scenario 2 investigates a more challenging case, where input models were trained with data from entirely different datasets, hence different distributions. To this end, we trained input models on COCO and Pascal-VOC datasets respectively. Similar to the previous scenario, we studied two choices of unlabeled data for our Model Composition method: a) unlabeled data from the same distribution as training data (in this case COCO+VOC images without using labels), and b) unlabeled data from a different dataset all together, e.g. the OID dataset. Note that the input models were trained on a different number of object categories (with overlap), and the output combined model was trained to support the union of object categories of the input models.

Table 3 shows the results of this experiment. It is observed from Table 3 that in the unsupervised case (i.e. no labeled data was used), our method achievs 29% and 27.4% mAP, close to the fully supervised performance of the upper-bound model. We also see from Table 3 that when partially labeled data are used for further fine-tuning, our method shows significant improvements over supervised training. In particular, when using 1%, 5%, and 10% of labels, our method shows +22.1%, +13%, and +10.3% gaps over supervised training.

Scenario 3: Combining input models with different architectures, that are trained on entirely different datasets. In this scenario, we studied the most generic case, in which input models have different architectures, are trained on different datasets, and with a different number of object categories. The output model also was chosen to have a different architecture than the input models (See Table 1). This scenario evaluated whether our method can combine the knowledge of models trained on different circumstances, data, and architecture, to a desired new and different architecture.

Table 3 shows the results of this experiment. It is observed from Table 3 that our method is very effective, and in some cases performs even better than supervised training with 100% of labels. When partial labels are available for fine-tuning, our method shows a strong performance, with large gaps compared to supervised training, especially in the low label range. Moreover, Table 3 shows that unsupervised training with our method achieved an mAP of 34%, only 1% below supervised training with all labels. After fine-tuning, we were able to meet the same performance as supervised training with only 10% of the original data.

Scenario 4: Having a large number of input models. This scenario investigates the case when a larger number of input models are provided. This would increase the diversity among the models since they can be trained on different data, or object categories, and thus results in a more challenging situation. To this end, we assumed 10 input models. Each model was trained on a randomly selected subset of the COCO dataset, so that training data for each model had no overlap to the other models. However, object categories could have overlap, as their type and count were chosen randomly. Supplementary materials [Banitalebi-Dehkordi et al.(2021)Banitalebi-Dehkordi, Kang, and Zhang] provides a visualization of the type and count of the object categories used for these 10 models. Since each model was trained with roughly 10% of the COCO training set, different number of object categories for different models resulted in a different per-class size of training data. The imbalance here made model composition harder, but mimicked realistic situations where training data can in fact be imbalanced for input models. As mentioned, for these 10 models, categories were randomly selected and the number of categories was selected from 5, 10, 20, 30, and 40. Note that generating 10$\times$ pseudo-labels on unlabeled data can be time-consuming (although it can be parallelized in production). Therefore, we only used 100K randomly selected examples from the OID dataset for this experiment.

We observe from Table 3 that our model composition method can effectively combine the 10 input models into a single new model with the union of their object categories.

Remark 3: A note on the unsupervised performance of OID: As observed in Table 3, in the challenging scenarios of 3 and 4, the unsupervised (0% labels) performance of model composition with OID^U is considerably lower that that of COCO^U or [COCO+VOC]^U. In this regard, there are a few points worth a mention:

• •

  In general, using unrelated arbitrary data is expected to result in a lower performance compared to using data from the same distribution as the input models’ train set, since pseudo-labels will be less reliable. This is exacerbated in challenging tasks such as scenario 3 where input models are trained on different data and have different architectures with respect to each other and the output model; or in scenario 4 where there are a large number of input models trained on different small-scale data.

• •

  It is worth reminding that the case of purely unsupervised model composition means combining an arbitrary number of black-box models (trained on arbitrary data with arbitrary architecture or categories), all without using any labels. In that sense, the real baseline to compare against is the supervised training, which performs much worse than model composition in low data regimes, even in the case of unrelated OID^U data.

• •

  Moreover, the main goal of the paper is to explore whether or not neural networks can be combined using only unlabeled data, and if yes, to what extent (hence the title). We observe from the results that the answer is for the most part yes; however, in case unlabeled data from the original distributions was not available, for some challenging scenarios a small percentage of labels may be needed to achieve a decent performance.

• •

  In a completely unsupervised setting, model composition can still effectively combine input models. The performance will be improved if the unlabeled set size is larger.

Remark 4: A note on practical applications: As mentioned in Section 3, a fundamental motivation for our work is a cloud services application, as shown in Figure 4, in which engineers and expert users can leverage a model composition service to build stronger models with combined skills, especially in the presence of a large variety of trained models and datasets on the cloud. Different scenarios in the experiments were also inspired by such a philosophy, but designed at various levels of difficulty. Here, we add a new practical use-case. In this new experiment, we combine separate models of face and mask detection, to build one that is suitable for both face & mask detection. Results are shown in Table 4.

Ablation on the number of categories: Next, we study the impact of the number of class categories in the performance. To this end, we take two input models from scenario 4, and combine them with a varying number of classes. $M_{1}$ is trained on 5 and $M_{2}$ is trained on 10 object categories (see Figure 9 in supplementary). Each time we add a number of random categories of $M_{2}$ to $M_{1}$, so the combined model can have 5,6,…,15 classes. Table 5 shows the results. Note that in each case the validation/training set will be different as it includes images of a particular set of categories. $M_{1}$ and $M_{2}$ have mAP of 27.1% & 25.4%, respectively (each has roughly 12K training, and 1K validation examples). In general, higher number of classes results in slightly lower mAP, but we should also note that unlabeled set size becomes also larger (i.e. more pseudo-labels).

In addition to our main results on the task of object detection, we also provide a highlight of our results on the task of image classification. Similar to object detection, we designed the classification experiments in the form of different scenarios.

In both scenarios, we tried model composition with unlabeled data from the Caltech-256 dataset (i.e. similar data distribution but without labels), and a 160K subset unlabeled data from OID (i.e. a different dataset altogether). Table 6 shows the results for these scenarios.

In Table 7, we provide a comparison between our method and two additional baselines: i) a simple model ensemble by aggregating directly the prediction of the input models; ii) knowledge distillation when using the input models as teachers, such as [Hinton et al.(2015)Hinton, Vinyals, and Dean, Ahn et al.(2019)Ahn, Hu, Damianou, Lawrence, and Dai]. For the second baseline, we consider the vanilla distillation [Hinton et al.(2015)Hinton, Vinyals, and Dean] but with soft labels.

It is observed from the results that the proposed method is effective in combining image classification models. In both the unsupervised and semi-supervised cases, our method performs competitively compared to supervised models, even when 100% of labels are used.

We share our implementation code to make it easy to reproduce our results. The source-code is attached to the supplementary materials in a ‘code’ directory. We also provide detailed instructions for training and evaluating our models in ‘README.md’ files.

Fig. 6 provides a visualization of object detection results of Table 3. We observe from this figure that in low data regimes model composition performs considerably better than supervised training with partial data. Fig. 7 shows an extended visualization on the cloud embodiment introduced in the paper. In this figure, we provide an easier comparison between before & after incorporating the model composition as a service. Moreover, Fig. 8 demonstrates an example of pseudo-label aggregation procedure of Algorithm 2. In addition, Fig. 9 visualizes the data splits of object detection scenario 4, where we combined 10 models trained on different COCO subsets.