Classifier Crafting: Turn Your ConvNet into a Zero-Shot Learner!

In this paper, we propose to entangle the problem of making the aforementioned “seen-to-unseen mapping” generalizable with the problem of learning ZSL-specific visual descriptors. We craft the weights $w_{i}$ of the softmax classifier’s of a standard convnet: that is, we keep $w_{i}$ freezed and we optimize by gradient descent the rest of the convnet accordingly. We propose two strategies for that. 1) Semantic crafting: we replace $w_{i}$ with the class embeddings which, thanks to Osherson’s default probabilities¹¹1E.g., we will give probability “1” for a zebra to be four legged and probability “0” for a flying otter. Note that these probabilities are not visually grounded [4] and only convey information on the expected appearance: not all images of a zebra will actually show four legs! [18] or CNN-LSTM text embeddings [35] can be casted into numerical vectors (that’s what $w_{i}$ are). 2) Visual crafting: we replace $w_{i}$ with visual prototypes. While seen prototypes are simply obtained by averaging visual features, unseen prototypes are inferred through a linear projection (optimized in closed-form) from class embeddings, in order not to violate the ZSL assumption. In both cases, we fine-tune the hierarchical feature representation of the convnet using seen images only, once the classifier is frozen after it has been visually or semantically crafted. Since our classification rules naturally extend towards unseen classes (albeit not requiring unseen data to be computed), we claim that the combination of them with a learnable and ZSL-tailored representation is a valuable proxy to generalize over unseen classes while being able to fine-tune a learnable representation only on the seen classes.

Since visual and semantic data are complementary in nature, we combine these two crafted convnets in an ensemble by simply averaging their predictions obtained from visually- and semantically-crafted softmax operators: in this manner, we improve prior art in standard, inductive ZSL [35, 29, 13, 19, 37, 32, 15, 8] – see Table 2.

To better deal with the GZSL setup, we propose a novel confidence rebalancing mechanism to fill the gap between seen predictions and (arguably) under-confident unseen predictions, given that unseen images are totally unobserved at training time. We do so by learning a discriminator to predict whether a vector of logits is seen (or not) using auxiliary task-irrelevant data to optimize it. Afterwards, we can use it to re-modulate, on the fly, the seen and unseen softmax probabilities at test time before taking a decision (again, neither training nor fine-tuning on unseen data is performed). By increasing the confidence of unseen classes and decreasing the ones of seen classes, (see Fig. 6), our rebalanced ensemble outperforms prior art in generalized, inductive ZSL [35, 29, 13, 19, 37, 32, 15, 8] – see Table 2.

We dissect the impact in performance of the visual and semantic crafting strategies, the rebalancing mechanism and the ensemble in ablation studies. We also provide grad-CAM [30] visualizations to gain in interpretability at no additional cost (see Fig. 1).

To summarize, these are the main contribution of our work:

• •

  We propose (visual and semantic) fixed classification rules to craft (i.e., replace and keep freezed) the softmax classifier’s weights. This turns a convnet into a zero-shot learner, Consequently, we learn from seen visual data only a feature representation which is tailored for ZSL and, thus, effective to recognize unseen classes as well.

• •

  With a simple ensemble method (average of predictions) our visual and semantic crafted networks improve prior art in standard, inductive ZSL. Once endowed of a novel predictions’ rebalancing mechanism, which we propose, our crafted networks improve improve prior art in inductive GZSL as well.

• •

  We provide ablation studies to quantify the role of each computational module in performance, comparing with prior rebalancing mechanisms [6] and [1]. We also gain interpretability simply using grad-CAM [30] as it is.

We first explain the proposed approach (Section 2) and present the supporting experimental evidences (Section 3). The discussion of related work is deferred to Section 4, in order to, hopefully, gain in readability. Conclusions will be drawn in Section 5.

We consider a convolutional feed-forward neural network (convnet) with a softmax classifier, trained in and end-to-end on raw images. Using the softmax, inference over a fixed pool of classes (indexed over $j$), can done through $\arg\max_{j}\mathbf{p}_{j}$ where

is the probability of predicting the $j$-th class²²2For efficiency issues, $\arg\max_{j}\mathbf{p}_{j}$ is usually replaced by $\arg\max_{j}\mathbf{f}\cdot w_{j}$ in standard deep learning libraries (e.g., PyTorch, MATLAB, Tensorflow, …), since both operations give the same result.. In Eq. (1), the classifier’s weights are denoted by $w_{j},\dots,w_{\ell},\dots$ (one weight $w_{j}$ per class) and $\mathbf{f}$ refers to the higher lever feature representation that is learnt (and fed to the classifier to compute the logits $\mathbf{f}\cdot w_{j}$).

We run experiments on AWA [17], CUB [33], SUN [36], FLO [24] datasets using the proposed splits “PS” which are ImageNet-compatible [34]. Class embeddings are extracted using the TF-IDF [13] for AWA, CUB, SUN and the LSTM-based approach of [35] for FLO. Standard performance metrics are used [34]: in standard, inductive ZSL performance is evaluated as mean top-1 classification accuracy ($T1$) over unseen classes. In generalized, inductive ZSL, we report $H$, the harmonic mean between mean per-class accuracy scores $S$ and $U$, computed over seen or unseen classes independently [34].

Our idea of crafting the classifier’s weights of a softmax operator to solve ZSL is broadly applicable to a number of deep convolutional networks, designed for image classification. In fact, in Fig. 7, we show how different networks – in this case, AlexNet [16], ResNet-101 [10], DenseNet-201 [12] and DarkNet-53 [28] (with 256 $\times$ 256 input layer) – can be turned into a classifier that is able to discriminate classes that remain unseen at training time (on FLO using the V&S-CC ensemble of visual and semantic crafting, we achieve $T1_{\rm AlexNet}=$ 67.45%, $T1_{\rm ResNet-101}=$ 77.51%, $T1_{\rm DenseNet-201}=$ 77.92% and $T1_{\rm DarkNet-53}=$ 78.10%).

We compare our proposed confidence rebalancing mechanism (Section 2.2) with two standard, state-of-the-art alternatives: calibrate stacking [6] and COSMO [1]. As the reader can see in Fig. 8 for FLO, our method is more effective than prior solutions [6, 1] in resolving the bias towards seen classes that are used to train S-CC, V-CC and V&S-CC models, making them effective to generalize onto unseen classes also.

On the same dataset, we provide a numerical ablation study to quantify the impact of performance of our rebalancing mechanism “R” (across all different crafting strategies) while also comparing with an “oracle” discriminator $\mathcal{O}$ which is always able to spot whether a test instance is seen or not: as shown in Tab. 1, the gap in between “$R$” and “$\mathcal{O}$” is only 1.8% for S-CC, increases to 6.6% for V-CC (but also the performance increases) and stabilizes to 8.1%. Albeit improvable, this gap certifies that the discriminator is effective enough in handling unseen classes and make predictions more balanced: after adding “$R$”, the performance in GZSL, measured through $H$, is always (more than) doubled.

For V-CC, the crafting rules $r_{i},\dots,\widetilde{r}_{k},\dots$ are obtained by averaging seen features extracted from a pre-trained convnet (and learning the prototypes for the unseen). Hence, we can still perform ZSL inference by simply crafting the classifier, while skipping any fine-tuning on the feature representation $\mathbf{f}$. By doing so, we can really evaluate the benefit of tailoring visual descriptors for ZSL: it turns out that this is indeed advantageous, since the fine-tuning free variant of V-CC is gapped in performance with respect to the full (i.e., fine-tuned) version (e.g., about -20% on CUB).

In the seminal work [18], ZSL is framed as the regression problem of predicting attributes from visual data. Differently, we exploit seen class embeddings and seen images to obtain better features with which we can do zero-shot inference on unseen classes, using a softmax classifier. Due to the absence of feature learning in [18], attributes’ predictions is highly suboptimal to our crafting - e.g., -32% on AWA and -30% on SUN). In CondZSL [20] and EXEM [5], classifiers explicitly depends upon class embeddings, using a conditional/causal perspective or exploiting data self-representation, respectively. However, neither [20] nor [5] perform feature learning from raw image data (they exploit pre-computed ImageNet-pretrained features) and, again they are suboptimal to us (V&S-CC always improve in performance both CondZSL nd EXEM, as one can check in Table 2).

End-to-end learnable zero-shot learning methods are emerging as recent directions for both video-based [3] (not considered here) or image-based applications [21]. However, in [21], the absence of a confidence rebalancing mechanism forbid the methods to be tested in GZSL - while in standard ZSL we are on par with [21] on AWA (see [21, Tab. 2]), superior on SUN (+2.3%) and sharply superior on CUB (+15%). Note that we did not insert [21] in Table 2 since it did not provide results using ResNet-101 features, using GoogleNet features instead. Note that, in fact, we could have furthermore improved our performance in Table 2 is using DarkNet-53 as opposed to ResNet-101 (e.g., about +3% improvement for $H$ on FLO), but this would not have been fair with respect to prior art. Ensemble approaches have been recently investigated in the (easiest) transductive ZSL setup - where unannotated visual data from unseen classes are accessible during training - [9, 38]. To the best of our knowledge we are the first of proposing an ensemble in inductive (standard and generalized) ZSL.

There are ZSL methods which, similarly to us, explicitly achieve interpretability using class attention maps: [7, 37]. Please, note that, in [7], an explicit patch-based optimization is designed, while [37] exploits body parts annotations (and not all ZSL datasets are provided of that and, usually this type of meta-data is not adopted). Our work is different since we achieve interpretability at not additional cost (no direct optimization for that, no extra annotation is required beyond class embeddings).

We briefly discuss methods which combine a fixed classifier with a learnable feature representation. There is a recent sequence of works [27, 11, 25, 26] applying this idea to diverse applications, namely, vanilla classification [11, 25], incremental learning [26] or few-shot learning. To the best of our knowledge, our work is the first one to progress this sequence, applying this paradigm in inductive zero-shot learning.

In [11, 25, 26], the classifier’s weights are fixed to a pre-defined geometrical configuration (for instance, mapping all classes to be recognized over the vertex of a simplex [25]). We cannot access all classes to be recognized at training time, but only the seen. Thus, we cannot adapt on unseen using newly incoming data from them (as in [26]) but we need to carry out predictions from zero shots. Hence, the classifier’s weights we craft cannot be explicitly dependent upon unseen data - and this is different from the few-shot learning approach of [27] in which data from all classes are used for this. We therefore opted for a solution in which we rely on the intrinsic seen-to-unseen transferrability of our fixed classification rules (used for crafting) so that the representation, that we learn by matching them, can be transferrable in turn.

As shown in Fig. 8. our proposed discriminator, predicting if a logit is seen or not while training on real seen logits and fake auxiliary unseen ones, seems valid in re-calibrating predictions in the generalized zero-shot learning regime (since enhancing the performance in GZSL quite sharply). Relatively to calibrate stacking [6] or confidence smoothing [1], it appears more effective in this respect. Beyond performance, we also technical differ from these works: our rebalancing is fully discriminative, while [6] is based on thresholding (by manually decreasing the predictor’s confidence) and [1] seeks for an adaptive probabilistic prior.