Revisiting Document Representations for Large-Scale Zero-Shot Learning

ZSL algorithms learn to align visual features and semantic representations using a set of seen classes $\mathcal{S}$. The alignment is then applied to the test images of unseen classes $\mathcal{U}$. We denote by $\mathcal{D}=\{({\bm{x}}_{n},y_{n}\in\mathcal{S})\}_{n=1}^{N}$ the training data (i.e., image feature and label pairs) with the labels coming from $\mathcal{S}$.

Suppose that we have access to a semantic representation $\bm{a}_{c}$ (e.g., word vectors) for each class $c\in\mathcal{S}\cup\mathcal{U}$, one popular algorithm DeViSE Frome et al. (2013) proposes the learning objective

	$\displaystyle\sum_{n}\sum_{c\neq y_{n}}\max\{0,\Delta$	$\displaystyle-f_{\bm{\theta}}^{\top}({\bm{x}}_{n})\bm{M}g_{\bm{\phi}}(\bm{a}_{y_{n}})$
		$\displaystyle+f_{\bm{\theta}}^{\top}({\bm{x}}_{n})\bm{M}g_{\bm{\phi}}(\bm{a}_{c})\},$		(1)

where $\Delta\geq 0$ is a margin. That is, DeViSE tries to learn transformations $f_{\bm{\theta}}$ and $g_{\bm{\phi}}$ and a matrix $\bm{M}$ to maximize the visual and semantic alignment of the same classes while minimizing that between classes. We can then classify a test image ${\bm{x}}$ by

$\displaystyle\operatorname{arg\,max}_{c\in\mathcal{U}}f_{\bm{\theta}}^{\top}({\bm{x}})\bm{M}g_{\bm{\phi}}(\bm{a}_{c}).$

(2)

Here, we consider that every class $c\in\mathcal{S}\cup\mathcal{U}$ is provided with a document $H_{c}=\{\bm{h}^{(c)}_{1},\cdots,\bm{h}^{(c)}_{|H_{c}|}\}$ rather than $\bm{a}_{c}$, where $|H_{c}|$ is the amount of sentences in document $H_{c}$ and $\bm{h}^{(c)}_{j}$ is the $j$th sentence, encoded by BERT Devlin et al. (2018). We mainly study DeViSE, but our approach can easily be applied to other ZSL algorithms.

We aim to filter out sentences in $H_{c}$ that are not describing visual information. We first leverage the section headers in Wikipedia pages, which indicate what types of sentences (visual or not) are in the sections. For example, the page “Lion” has sections “Description” and “Colour variation” that are likely for visual information, and “Health” and “Cultural significance” that are for non-visual information.

To efficiently identify these section headers, we use ImageNet synsets Deng et al. (2009), which group objects into $16$ broad categories. We randomly sample $30\sim 35$ classes per group, resulting in a set of $500$ classes. We then retrieve the corresponding Wikipedia pages by their names and manually identify section headers related to visual sentences. By sub-sampling classes in this way, we can quickly find section headers that are applicable to other classes within the same groups. Table 1 shows some visual/non-visual sections gathered from the 500 classes. For example, “Characteristics” frequently appears in pages of animals to describe their appearances. In contrast, sections like “History” or “Mythology” do not contain visual information. Investigating all the $500$ Wikipedia pages carefully, we find $40$ distinct visual sections. We also include the first paragraph of a Wikipedia page, which often contains visual information.

Our second approach uses K-means for sentence clustering: visual sentences often share common words and phrases of visual attributes, naturally forming clusters. We represent each sentence using the BERT features Devlin et al. (2018), and perform K-means (with $K=100$) over all the sentences from Wikipedia pages of ImageNet classes. We then manually check the 100 clusters and identify $40$ visual clusters. Table 2 shows a visual (top) and a non-visual (bottom) cluster. We highlight sentences related to two classes: “kit-fox” (red) and “tiger” (blue). The visual cluster describes the animals’ general appearances, especially about visual attributes “dark”, “black”, “tail”, “large”, etc. In contrast, the non-visual cluster describes mating and lifespan that are not related to visual aspects.

After we obtain a filtered document $\hat{H}_{c}$, which contains sentences of the visual sections and clusters, the next step is to represent $\hat{H}_{c}$ by a vector $\bm{a}_{c}$ so that nearly all the ZSL algorithms can leverage it.

A simple way is average, $\bar{\bm{a}}_{c}=\frac{1}{|\hat{H}_{c}|}\sum_{\bm{h}\in\hat{H}_{c}}\bm{h}$, where $\bm{h}$ is the BERT feature. This, however, may not be discriminative enough to differentiate similar classes that share many common descriptions (e.g., dog classes share common phrase like “a breed of dogs” and “having a coat or a tail”).

We therefore propose to identify informative sentences that can enlarge the difference of $\bm{a}_{c}$ between classes. Concretely, we learn to assign each sentence a weight $\lambda$, such that the resulting weighted average $\bm{a}_{c}=\frac{1}{|\hat{H}_{c}|}\sum_{\bm{h}\in\hat{H}_{c}}\lambda(\bm{h})\times\bm{h}$ can be more distinctive. We model $\lambda(\cdot)\in\mathbb{R}$ by a multi-layer perceptron (MLP) $b_{\bm{\psi}}$

$\displaystyle\lambda(\bm{h})=\frac{\exp(b_{\bm{\psi}}(\bm{h}))}{\sum_{{\bm{h}}^{\prime}\in\hat{H}_{c}}\exp(b_{\bm{\psi}}({\bm{h}}^{\prime}))}.$

(3)

We learn $b_{\bm{\psi}}$ to meet two criteria. On the one hand, for very similar classes $c$ and $c^{\prime}$ whose similarity $\cos(\bm{a}_{c},\bm{a}_{c^{\prime}})$ is larger than a threshold $\tau$, we want $\cos(\bm{a}_{c},\bm{a}_{c^{\prime}})$ to be smaller than $\tau$ so they can be discriminable. On the other hand, for other pair of less similar classes, we want their similarity to follow the average semantic representation $\bar{\bm{a}}_{c}$²²2The purpose of introducing $\lambda(\cdot)$ is to improve $\bm{a}_{c}$ from the average representation $\bar{\bm{a}}_{c}$ to differentiate similar classes..

To this end, we initialize $b_{\bm{\psi}}$ such that the initial $\bm{a}_{c}$ is close to $\bar{\bm{a}}_{c}$. We do so by first learning $b_{\bm{\psi}}$ to minimize the following objective

$\displaystyle\sum_{c\in\mathcal{S}\cup\mathcal{U}}\max\{0,\epsilon-\cos(\bm{a}_{c},\bar{\bm{a}}_{c})\}.$

(4)

We set $\epsilon=0.9$, forcing $\bm{a}_{c}$ and $\bar{\bm{a}}_{c}$ of the same class to have $\cos(\bm{a}_{c},\bar{\bm{a}}_{c})>0.9$. We then fine-tune $b_{\bm{\psi}}$ by minimizing the following objective

$\displaystyle\sum_{c}^{\mathcal{S}\cup\mathcal{U}}\sum_{c\neq c^{\prime}}^{\mathcal{S}\cup\mathcal{U}}\max\{0,\cos(\bm{a}_{c},\bm{a}_{c^{\prime}})-\tau\}.$

(5)

We assign $\tau$ a high value (e.g., $0.95$) to only penalize overly similar semantic representations. Please see the appendix for details.

We use the ImageNet Fall 2011 dataset Deng et al. (2009) with $21,842$ classes. We use the 1K classes in ILSVRC 2012 Russakovsky et al. (2015) for DeViSE training and validation (cf. Equation 1), leaving the remaining $20,842$ classes as unseen classes for testing. We follow Changpinyo et al. (2016) to consider three tasks, 2-Hop, 3-Hop, and ALL, corresponding to 1,290, 5,984, and 14,840 unseen classes that have Wikipedia pages and word vectors and are within two, three, and arbitrary tree hop distances (w.r.t. the ImageNet hierarchy) to the 1K classes. On average, each page contains 80 sentences. For images, we use the $2,048$-dimensional ResNet visual features He et al. (2016) provided by Xian et al. (2018a). For sentences, we use a $12$-layer pre-trained BERT model Devlin et al. (2018). We denote by BERT_p the pre-trained BERT and BERT_f the one fine-tuned with DeViSE. Please see the appendix for details.

Word vectors of class names are the standard semantic representations for ImageNet. Here we compare to the state-of-the-art w2v-v2 provided by Changpinyo et al. (2020), corresponding to a skip-gram model Mikolov et al. (2013) trained with ten passes of the Wikipedia dump corpus. For ours, we compare using all sentences (NO), visual sections (Vis_sec) or visual clusters (Vis_clu), and both (Vis_sec-clu). On average, Vis_sec-clu filters out 57$\%$ of the sentences per class. We denote weighted average (Section 3.4) by BERT_p-w and BERT_f-w.

The original DeViSE Frome et al. (2013) has $f_{\bm{\theta}}$ and $g_{\bm{\phi}}$ as identity functions. Here, we consider a stronger version, DeViSE^⋆, in which we model $f_{\bm{\theta}}$ and $g_{\bm{\phi}}$ each by a two-hidden layers multi-layer perceptron (MLP). We also experiment with two state-of-the-art ZSL algorithms, EXEM Changpinyo et al. (2020) and HVE Liu et al. (2020).

We use the average per-class Top-1 classification accuracy as the metric Xian et al. (2018a).

Table 3 summarizes the results on ImageNet. In combining with each ZSL algorithm, our semantic representations Vis_sec-clu that uses visual sections and visual clusters for sentence extraction outperforms w2v-v2. More discussions are as follows.

Table 4 summarizes the results on AwA2 Xian et al. (2018a) and aPY Farhadi et al. (2009). The former has $40$ seen and $10$ unseen classes; the latter has $20$ seen and $12$ unseen classes. We apply DeViSE together with the $2,048$-dimensional ResNet features He et al. (2016) provided by Xian et al. (2018a). Our proposed semantic representations (i.e., BERT_p + Vis_sec-clu) outperform w2-v2 and the manually annotated visual attributes on both the ZSL and generalized ZSL (GZSL) settings. Please see the appendix for the detailed experimental setup. These improved results on ImageNet, AwA2, and aPY demonstrate our proposed method’s applicability to multiple datasets.

To further justify the effectiveness of our approach, we compare to additional baselines in Table 5.

• •

  BERT_p-w-direct: it directly learns $b_{\psi}$ (Equation 3) as part of the DeViSE objective. Namely, we directly learn $b_{\psi}$ to identify visual sentences, without our proposed selection mechanisms, such that the resulting $\bm{a}_{c}$ optimizes Equation 1.

• •

  Par_1st: it uses the first paragraph of a document.

• •

  Cls_name: it uses the sentences of a Wikipedia page that contain the class name.

As shown in Table 5, our proposed sentence selection mechanisms (i.e., Vis_sec, Vis_clu, and Vis_sec-clu) outperform all the three baselines.