Task-Adaptive Negative Envision for Few-Shot Open-Set Recognition

We start with a simple generator that consists of a single MLP layer applied on averaged class prototypes, i.e.,

$$\mathbf{p}^{-}={f_{n}}(\mathbf{p}_{avg}^{-}),~{}~{}\mathbf{p}_{avg}^{-}=\frac{1}{N}\sum\nolimits_{c\in\mathcal{C}^{f}}\mathbf{p}_{c}$$

(3)

where ${f_{n}}$ is a MLP that takes $\mathbf{p}_{avg}^{-}$, the mean of $\mathbf{P}^{f}$, as input so that $\mathbf{p}^{-}$ is independent from the prototype order. Meanwhile, we set $\mathbf{p}_{avg}^{-}$ as a naive baseline (AVG) as $\mathbf{p}_{avg}^{-}$ is also order-independent.

Transformers[42] are proved effective in exploiting relations, which is also independent from the input order (without positional encoding). We apply a standard Transformer attention block over few-shot class prototypes to generate the negative prototype. Specifically, we calculate the self-attention weight between class prototypes, i.e.,

$$\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{f})}=\frac{1}{\sqrt{d}}(\mathbf{P}^{f}\mathbf{K}_{q}(\mathbf{P}^{f}\mathbf{K}_{k})^{T}),$$

where $\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{f})}\in\mathcal{R}^{|\mathcal{C}^{f}|\times|\mathcal{C}^{f}|}$ is the attention weights matrix, and $\mathbf{K}_{q},\mathbf{K}_{k}\in\mathcal{R}^{d\times d}$ are trainable linear projection kernels. Then, we normalize the weights and output

$$\mathbf{P}^{{}^{\prime}}=\mathbf{P}^{f}+\sigma(\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{f})})(\mathbf{P}^{f}\mathbf{K}_{v}),$$

where $\sigma(\cdot)$ is a softmax function for each row in $\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{f})}$ and $\mathbf{K}_{v}\in\mathcal{R}^{d\times d}$ is another trainable linear projection kernel. Then, we feed the average of $\mathbf{P}^{{}^{\prime}}$ to a MLP function ${f_{n}}$ to get the negative prototype $\mathbf{p}^{-}$.

The above generators is suitable for FSOR problem. Now we consider the more challenging GFSOR task. Directly employing the above methods may introduce bias towards $\mathcal{C}^{*}$ as $\mathcal{C}^{*}$ has plenty of training samples and the prototype $\mathbf{P}^{*}$ can be better-estimated compared against the few-shot prototypes $\mathbf{P}^{f}$. Hence we need another negative generator compatible with GFSOR , which should take care of both $\mathcal{C}^{*}$ and $\mathcal{C}^{f}$. We build our ATT-G generator on top of a popular GFSL method [10], which uses an attention mechanism to calibrate few-shot prototypes $\mathbf{P}^{f}$ with $\mathbf{P}^{*}$. Specifically, we follow [10, 9, 41] to first train a network under large-scale classification task using the labeled samples of $\mathcal{C}^{*}$ (i.e., pre-training) and use the weight in the last linear layer as many-shot class prototypes $\mathbf{P}^{*}$. Then we apply the attention block between $\mathbf{P}^{f}$ and $\mathbf{P}^{*}$ to generate the negative prototype $\mathbf{p}^{-}$, i.e.,

$$\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{*})}=\frac{1}{\sqrt{d}}(\mathbf{P}^{f}\mathbf{K}_{q}(\mathbf{P}^{*}\mathbf{K}_{k})^{T}),$$

(4)

$$\mathbf{P}^{{}^{\prime}}=\mathbf{P}^{f}+\sigma(\mathbf{A}_{(\mathbf{P}^{f},\mathbf{P}^{*})})(\mathbf{P}^{*}\mathbf{K}_{v}),$$

(5)

and $\mathbf{p}^{-}$ is similarly computed by feeding the average of $\mathbf{P}^{{}^{\prime}}$ into a MLP $f_{n}$. Furthermore, we’d like to filter out task-irrelevant information by applying a channel-wise gating mechanism on top of $\mathbf{P}^{f}$:

$$\mathbf{p}_{c}^{\prime}=\mathbf{p}_{c}\odot\phi({f_{g}}(\frac{1}{N-1}\sum\nolimits_{i\in\mathcal{C}^{f}\setminus\{c\}}\mathbf{p}_{i})),$$

(6)

for $c\in\mathcal{C}^{f}$ where $\odot$ and $\phi$ denote element-wise multiplication and sigmoid operation, and ${f_{g}}$ is a fully-connected layer. Then, we use the updated $\mathbf{P}^{f^{\prime}}$ to replace the input $\mathbf{P}^{f}$ in Eq. 4. Finally, we follow the order of many-shot prototypes $\mathbf{P}^{*}$, few-shot prototypes $\mathbf{P}^{f}$, and negative prototype $\mathbf{p}^{-}$ to build the open-set classifier for a GFSOR task, where $\mathbf{P}^{*}$ are the weights in the last linear layer after pre-training.

Inspired by recent cross-modal FSL works [22, 45], we further explore how class semantics could help model negative class. Specifically, we use a cross-modal attention mechanism on top of ATT-G. For each class $c\in\mathcal{C}^{f}\cup\mathcal{C}^{*}$, we concatenate $\mathbf{p}_{c}$ with its word embedding $\mathbf{e}_{c}\in\mathcal{R}^{w}$ along the channel to have $\mathbf{z}_{c}=[\mathbf{p}_{c},\mathbf{e}_{c}]\in\mathcal{R}^{w+d}$. Then we use $\mathbf{Z}^{f}$ and $\mathbf{Z}^{*}$ instead of $\mathbf{P}^{f},\mathbf{P}^{*}$ in Eq. 4 to calculate the attention. And stick to Eq. 5 to take $\mathbf{P}^{*}$ as input since we are still comparing visual features for recognition.

In addition, we can easily extend from single negative generation to multiple negative generation. Specifically, we can learn a set of generators $\{g_{n,i}\}_{i=1}^{M}$ to generate multiple negative prototypes for each task. For ATT, ATT-G, and SEMAN-G, to reduce the number of trainable parameters, we choose to share the linear projection kernels in attention mechanism used to calculate $\mathbf{P}^{{}^{\prime}}$, but just train separate MLPs $\{f_{n,i}\}_{i=1}^{M}$ to synthesis multiple negative prototypes. In this way, we get multiple thresholds $\{\theta_{a,i}\}_{i=1}^{M}$. Then the maximum threshold $\theta_{a}=\text{max}\big{(}\{\theta_{a,i}\}_{i=1}^{M}\big{)}$ is used as the final threshold for open-set recognition.

Standard FSOR meta-training strategy [24, 37] trains the model by simulating FSOR tasks from the given base dataset $\mathcal{D}^{B}$. Specifically, it trains on a set of tasks sampled from the base dataset $\mathcal{D}^{B}$ where the images are from base classes $\mathcal{C}^{B}$. Within an $N$-way $K$-shot FSOR task $\mathcal{T}$, unknown sources are simulated using a different set of $N$ classes $\mathcal{C}^{n}$, i.e., $\mathcal{C}^{n}\subset\mathcal{C}^{B}-\mathcal{C}^{f}$ where $|\mathcal{C}^{n}|=N$, and then randomly sample images belonging to $\mathcal{C}^{n}$ from $\mathcal{D}^{B}$ for $\mathcal{Q}^{n}$. Then the model is trained using an objective, typically an open recognition loss within the sampled $\mathcal{T}$. The standard FSOR meta-training can be generalized to GFSOR. For a GFSOR task $\mathcal{T}^{*}$, we can sample $\mathcal{C}^{f}$ and $\mathcal{C}^{*}$ from $\mathcal{C}^{B}$ and then simulate unknown sources as $\mathcal{C}^{n}\subset\mathcal{C}^{B}-(\mathcal{C}^{f}\cup\mathcal{C}^{*})$ where $|\mathcal{C}^{n}|=N$. And a GFSOR training objective may be specified to learn the model for GFSOR. Note that, during inference time (i.e., meta-testing), tasks are sampled from novel dataset $\mathcal{D}^{N}$ where the images are from classes $\mathcal{C}^{N}$ and no sample from $\mathcal{C}^{N}$ is seen during meta-training, i.e., $\mathcal{C}^{B}\cup\mathcal{C}^{N}=\varnothing$.

The idea of conjugate training is to sample task pairs whose few-shot examples of one task are used as the negative source of the other. Formally, we define two tasks $\mathcal{T}_{1}=(\mathcal{S}_{1},\mathcal{Q}_{1}^{f},\mathcal{Q}_{1}^{n}|\mathcal{C}^{f}_{1})$ and $\mathcal{T}_{2}=(\mathcal{S}_{2},\mathcal{Q}_{2}^{f},\mathcal{Q}_{2}^{n}|\mathcal{C}^{f}_{2})$ as a conjugate task pair when $\mathcal{Q}_{1}^{n}=\mathcal{Q}_{2}^{f}$ and $\mathcal{Q}_{1}^{f}=\mathcal{Q}_{2}^{n}$, i.e., the few-shot classes $\mathcal{C}^{f}_{1}$ ($\mathcal{C}^{f}_{2}$) in $\mathcal{T}_{1}$($\mathcal{T}_{2}$) is used as the negative source in task $\mathcal{T}_{2}$($\mathcal{T}_{1}$). For a conjugate GFSOR task pair ($\mathcal{T}^{*}_{1},\mathcal{T}^{*}_{2}$), in addition, $\mathcal{T}^{*}_{1}$ and $\mathcal{T}^{*}_{2}$ share the same many-shot class $\mathcal{C}^{*}$ and its queries $\mathcal{Q}^{*}$.

We use a standard cross-entropy loss $\mathcal{L}_{CE}(\cdot,\cdot)$ [26]. For an FSOR task $\mathcal{T}$, we use cosine similarity as $f_{s}$ and use $\mathcal{Q}^{f}\cup\mathcal{Q}^{n}$ to perform $(N+1)$-way classification. For each positive query $q\in\mathcal{Q}^{f}$, we learn to maximize the class score of its label category by minimizing $\mathcal{L}_{CE}(y_{q},q)$ where $y_{q}\in\{1,...,N\}$ is the class label of $q$. For each negative query $q\in\mathcal{Q}^{n}$, we set its ground truth label as $N+1$ and maximize the threshold $\theta_{a}$ by minimizing $\mathcal{L}_{CE}(N+1,q)$. During conjugate training, we consider the dependency of negative sampling mentioned in [20]. Without loss of generality, for a positive query $q\in\mathcal{Q}_{2}^{f}$ belonging to class $c_{n}\in\mathcal{C}^{f}_{2}$ in $\mathcal{T}_{2}$, it is used as the negative query and is trained to have high similarity with the negative prototype in $\mathcal{T}_{1}$.

With a simple classification loss, the negative prototypes are optimized to learn a tight rejection boundary for a specific task. Besides, for the attention-based generators, we also regularize the intermediate variables $\mathbf{P}^{{}^{\prime}}$ as class-specific negative prototypes. For each prototype $\mathbf{p}_{c}^{\prime}\in\mathbf{P}^{{}^{\prime}}$ generated from a positive prototype $\mathbf{p}_{c}$, we can think of $\mathbf{p}_{c}^{\prime}$ as the negative prototype for class $c$. Then, for each $\mathbf{p}_{c}^{\prime}$, we minimize its similarity with queries of class $c$ and maximize its similarity of negative queries with a binary cross-entropy loss $\mathcal{L}_{BCE}$

	$\displaystyle\mathcal{L}_{neg}(c)$	$\displaystyle=\frac{1}{|\mathcal{Q}_{c,1}^{f}|}\sum\nolimits_{q\in\mathcal{Q}_{c,1}^{f}}\mathcal{L}_{BCE}(0,f_{s}(f(q),\mathbf{p}_{c}^{\prime}))$
		$\displaystyle+\frac{1}{|\mathcal{Q}_{1}^{n}|}\sum\nolimits_{q\in\mathcal{Q}_{1}^{n}}\mathcal{L}_{BCE}(1,f_{s}(f(q),\mathbf{p}_{c}^{\prime}))$

where $\mathcal{Q}_{c,1}^{f}=\{q|q\in\mathcal{Q}_{1}^{f},y_{q}=c\}$ and $y_{q}$ denotes the class label of $q$. Finally, without loss of generality, for $\mathcal{T}_{1}$ in the conjugate task pair ($\mathcal{T}_{1},\mathcal{T}_{2}$), we have

$$\mathcal{L}_{\mathcal{T}_{1}}=\mathcal{L}_{CE}(\mathcal{Q}_{1}^{f}\cup\mathcal{Q}_{1}^{n})+\frac{1}{|\mathcal{C}_{1}^{f}|}\sum\nolimits_{c\in\mathcal{C}_{1}^{f}}\mathcal{L}_{neg}(c),$$

(7)

and the total conjugate training loss is $\mathcal{L}=\mathcal{L}_{\mathcal{T}_{1}}+\mathcal{L}_{\mathcal{T}_{2}}$.

Similarly, for the network trained on GFSOR tasks, given a conjugate task pair ($\mathcal{T}^{*}_{1},\mathcal{T}^{*}_{2}$), we have

$$\mathcal{L}_{\mathcal{T}^{*}_{1}}^{*}=\mathcal{L}_{CE}(\mathcal{Q}_{1}^{*}\cup\mathcal{Q}_{1}^{f}\cup\mathcal{Q}_{1}^{n})+\frac{1}{|\mathcal{C}_{1}^{f}|}\sum\nolimits_{c\in\mathcal{C}_{1}^{f}}\mathcal{L}_{neg}(c),$$

where the class label for a negative query is $N+1+|\mathcal{C}^{*}|$, and the total loss being $\mathcal{L}^{*}=\mathcal{L}_{\mathcal{T}^{*}_{1}}^{*}+\mathcal{L}_{\mathcal{T}^{*}_{2}}^{*}$. In this way, our conjugate training involves the class-correlation during network training.

We first compare different choices of negative generators on FSOR tasks in Tab. LABEL:tab:neg_gen. Note that ATT-G and SEMAN-G can also be applied for FSOR and compared to other approaches since all models are trained using base set only (including base prototype) and doesn’t use any extra data. We can see that attention-based methods are effective in negative generation as they are good at modeling inter-class relations. Adding class semantic information is also beneficial for discrimination. Meanwhile, by enabling multiple negative prototypes, i.e., $M=5$, we can automatically estimate the threshold $\theta_{a}$ with more flexibility, which then achieve consistent performance gain in F1-score, when compared generating a single negative prototype $M=1$. For the following experiment results, we set $M=5$ for our methods.

For threshold-based methods, threshold-tuning is crucial to get good recognition performance. To evaluate the overall open-set recognition, we compare macro-weighted F1-score. For threshold-based approaches, we define different thresholds and compute the corresponding F1-score. We illustrate our result in both Tab. LABEL:tab:neg_gen and Fig. 3(a), where we consider two threshold-based classifiers PEELER[24] and a combination of PEELER and our ATT-G method baseline, Dynamic[10], which calibrates novel prototype with base-class prototypes. In detail, we apply PEELER’s open-set training strategy on top of Dynamic.

In Fig. 3(a), we simulate 45k FSOR tasks and find their optimal rejection thresholds $\theta_{m}$ for the threshold-based approach Dynamic+PEELER. We plot the distribution of $\theta_{m}$, which shows that it covers a wide range between $0$ and $1$. It demonstrates that different FSOR tasks may need very different rejection threshold in practice with current threshold-based approach. And the overall recognition performance largely depends on threshold selection, as is shown in Tab. LABEL:tab:neg_gen. Our method instead automatically learn a task-adaptive rejection boundary, and we can see from Tab. LABEL:tab:neg_gen that all our negative envision instantiations outperform threshold-based methods. Fig. 3(b) further analyze the recognition behaviour under different openness [39]:

$$\text{openness}=1-\sqrt{\frac{2|\mathcal{C}^{f}|}{2|\mathcal{C}^{f}|+|\mathcal{C}^{n}|}}$$

where we fix $|\mathcal{C}^{f}|=5$ and vary $|\mathcal{C}^{n}|$ from $5$ to $15$. Similarly, we test for $600$ randomly selected FSOR tasks and take the average. As is validated by Fig. 3, our method clearly outperforms threshold-based methods at all openness levels.

We compare our method with other SOTA methods. The baselines we compare to include standard FSL methods (ProtoNet, FEAT), large-scale OR methods (OpenMax, CounterFactual), and existing FSOR methods (PEELER, SnaTCHer). We cite most of the baseline results from [20], and additionally compare to CounterFactual, a generative OR method which synthesize fake negative images and then train a $N+1$ classifier. To apply in our FSOR setting, we first train its GAN network on base set and use the support set to synthesize fake images. The averaged fake image feature is used as the negative prototype for FSOR.

Tab. LABEL:tab:fsor_sota demonstrates the results. Standard FSL methods perform poorly in negative detection due to its closed-set nature. Large-scale OR methods yields unsatisfactory performance especially on 1-shot classification. Interestingly, CounterFactual gives a relatively fair performance on negative detection, which also validates our concept of negative envision. But it’s still much worse than our few-shot-specific negative generation strategy, which validates that our approach better suits for the limited data scenario. Both ATT-G and SEMAN-G outperform other methods on Mini-ImageNet and get comparable result on Tiered-ImageNet.

Tab. LABEL:tab:fsor_ablate shows the impact of conjugate training. We observe consistent improvement on all metrics and datasets, validating that conjugate training efficiently boosts the learning process by enabling mutual supervision from two tasks.