The poll sampler aims to obtain a fine feature set $\mathbb{F}_{f}$. Since explicitly learning a binary sampler is infeasible, we develop a sample as ranking strategy. We use a small meta-scoring network to predict the informativeness score for each spatial feature location $(i,j)$:

The larger the score is, the more informative the feature vector $\textbf{f}_{ij}$ is. We then sort all the scores $\{s_{ij}\}$ as

where $\aleph$ is the sorting order and $L=HW$. With $\aleph$, we then take the top $N$ scoring vectors to form the fine feature set:

To enable the learning of $\mathrm{ScoringNet}$ with back-propagation, we take the predicted informativeness score as a modulating factor to the sampled fine feature set:

We find that normalizing the feature vectors before modulating can stabilize the learning of $\mathrm{ScoringNet}$:

We use layer normalization [1] and turn off the affine parameters. Ideally, $N$ may vary with the image content, but we observe that fixed amount sampling already generates good performance, i.e., $N=\alpha L$ where $\alpha$ is a constant fractional value, which we name as the poll ratio. This design also enables an extension to single model computation and performance trade-off discussed in Sec. 3.4.

The above poll sampler extracts the fine feature set. The remaining feature vectors mainly correspond to the background area. To compress them into a small feature set that summarizes the contextual information, we design a pool sampler that performs a weighted pooling of the remaining feature vectors to obtain a fixed number of $M$ background contextural feature vectors. This is partially inspired by the bilinear pooling [19] and double attention [4] operation where global descriptors are generated for capturing the second-order statistics of the feature map. Formally, the remaining feature vector set is

We project the feature vectors with a learnable weight $\textbf{W}^{a}\in\mathbb{R}^{C\times M}$ to obtain the aggregation weight $\textbf{a}_{r}\in\mathbb{R}^{M}$ :

and project the feature vectors with a learnable weight $\textbf{W}^{v}\in\mathbb{R}^{C\times C}$ to obtain the projected feature:

We then normalize the aggregation weight over all the remaining non-sampled locations with softmax:

With the normalized aggregation weight, the projected feature vectors are aggregated to obtain a new feature vector that summarizes the information of non-sampled locations:

By aggregating with all $M$ aggregation weights, we obtain the summarized coarse background contextual feature set:

It has been shown in [34] that the context information is crucial for recognizing the objects and is better aggregated by pyramid features of different scales. Our pool sampler is able to freely obtain context information of different scales, by dynamically generating the aggregation weights. That is, some feature vectors may capture local context while others may encode global context. We empirically show such an ability of the pool sampler by visualizing the aggregation weights. Together with the fine set $\mathbb{F}_{f}$ from the poll sampler, the desired abstraction set $\mathbb{F}^{*}$ is obtained. Note that instead of convolution feature map, the PnP module can also be applied after a transformer layer.

The PnP module reduces the image feature map from 2D coordinate space to an abstracted space, which cannot be used for dense prediction tasks like image segmentation. To address the limitation, we propose to project the encoder output feature vectors back to the 2D coordinate space. Specifically, the fine feature vectors are scattered back to the sampled locations; the coarse feature vectors are first diffused back to original 2D space with the aggregation weight:

and then scattered back the non-sampled locations of the poll sampler. $\hat{\textbf{f}_{m}}$ denotes output coarse feature vector from the encoder and $\hat{\textbf{f}}_{r}$ means the projected feature vector. The obtained 2D feature map is then used for dense prediction.

Recently [7] introduced a transformer-based image recognition model named Vision Transformer (ViT). We evaluate the generalizability of our method on the ViT model. We build the PnP-DETR and PnP-ViT by plugging the PnP module before the transformer network. The resulting models are end-to-end learned and other settings are the same with original models. We use the hybrid ViT architecture [7]. Unlike original DETR and ViT wherein the transformer directly operates over the full image feature space, the PnP augmented transformer models the information interaction on the fine-coarse feature space and adaptively allocates its computation in the spatial domain to achieve better efficiency.

To achieve different computation and performance trade-offs, existing methods improving transformer efficiency generally train multiple models with different complexities controlling hyperparameters, e.g., number of hashes in Reformer [16] and projected feature dimension in Linformer [29]. Unlike them, a model equipped with a PnP module can achieve instant single model computation and performance trade-off. This is enabled by controlling the poll ratio $\alpha$ to determine the amount of fine information preserved. With a larger $\alpha$, more fine feature vectors are obtained, and the overall performance is expected to be higher; with a smaller $\alpha$, the performance may be lower but more computation is saved. However, we find inference with a different $\alpha$ to training severely degrades the performance. We propose to generate a random poll ratio during training:

Where $\alpha_{low}$ and $\alpha_{high}$ defines the value range. $\alpha$ is updated in each iteration. In this way, the transformer learns to work with variable length of input feature vectors, and thus achieves the desired single model computation and performance trade-off by inferring with different poll sample ratios (Fig. 3). The model only needs to be trained once.

Tab. 1 shows the results of the fixed poll ratio training on the COCO benchmark. For the DETR-R50 model, with an $\alpha$ = 0.33, PnP-DETR achieves 41.1 AP and 60% reduction of transformer computation cost. Further increasing $\alpha$ to 0.5, the performance reaches a similar level as the DETR baseline (AP of 41.8 vs. 42.0), with 45% reduction of the computation. For DETR-R50-DC5 model, a similar trend is observed but more computation is saved. We also evaluate the setting of mismatched training and test poll ratio. The model trained with $\alpha$ = 0.33 gets nearly 5 AP drop when evaluating with $\alpha$ = 0.5. This observation shows the necessity of applying random poll ratio training for the model to work with variable poll ratio. We also compare to the deformable DETR [37], as we did not incorporate multi-scale features, which is not the focus of this work, we compare to single scale deformable DETR for fair comparison. Our method performs better than deformable DETR with less FLOPs, especially for large objects, e.g., AP_l of 60.0 vs. 57.8 for the ResNet-50 backbone.

As shown in Fig. 4, by training with the random poll ratio with a value range of $(0.15,0.8)$, the obtained model can achieve dynamic computation and performance trade-off by evaluating with variable poll ratio. The AP for certain poll ratio is similar to the fixed poll ratio trained counterpart. For example, a PnP-DETR-R50 model gets 41.1 AP with fixed poll ratio 0.33 training and 41.2 AP with random poll ratio training. The performance is the same to the baseline with a poll ratio of 0.65. We observe when the poll ratio is large, e.g., 0.5, increasing the poll ratio brings diminished gain in AP. This is likely because the fine feature set already covers the essential spatial locations for detecting the objects, and thus more fine information only brings limited gain. Similar observations are made with the ResNet-101 backbone. Tab. 2 shows the inference time compared to baseline model, the inference time is significantly reduced.

Fig. 5 shows some example detection results and associated computation density maps, with poll ratio of 0.33. The objects are well detected while the computation is dynamically allocated to the spatial domain in a content-aware manner. To compute the density map, we assign a weight to each spatial location. For poll sampled locations, the weight is 1. For each of other locations, the weight is the cumulative value of all pool sample aggregation weights at this location. Then the transformer cost is distributed with the normalized weights to obtain the computation density map.

Following [3], we evaluate our method on the panoptic segmentation task. To perform dense per-pixel segmentation as DETR, we project the encoder output feature back to the original 2D coordinate space. As shown in Tab. 3, the model saves computation and achieves instant performance and computation trade-off by varying the poll ratio $\alpha$, e.g., achieving Panoptic Quality (PQ) of 43.2 compared to 43.4 of a baseline DETR model, with 5G less FLOPs (i.e., 6.6G vs. 11.6G).

We also apply the PnP sampling to the recent transformer-based image classification model of ViT [7]. We use the hybrid architecture with ResNet50-stage4 feature map (14x14) and train the model on the ImageNet-1k dataset from scratch. We set the pool sample number to 10. We train the PnP-ViT with random poll ratio in the value range of $[0.2,0.8]$. As shown in Tab. 4, the PnP-ViT achieves dynamic computation and performance trade-off as observed with the DETR model. The results show the generalizability of PnP sampling design.

As shown in Fig. 6, we vary the the poll sample ratio and the pool sample number to obtain the performance curve with the same amount of computation cost. We observe that 1) with only poll sampling ($\alpha$-0.4), the performance is suboptimal; incorporating pool feature vector samples can significantly improve AP with the complementary background information from non-sampled locations, e.g., $\alpha$-0.39-$M$-10 model achieving about 0.7 AP higher than the $\alpha$-0.4 model. 2) with only pool sampling, the performance drops by a large margin. We assume it is difficult for the pool sampling to preserve accurate fine information, as it is designed to aggregate feature vectors spatially from different locations. 3) the optimal setting is $1/3$ poll ratio with $60$ pool samples, indicating that a compact feature set should be mainly composed of fine feature vectors for accurate object detection. We further individually examine the effects of pool sample number $M$ and poll sample ratio $\alpha$: 1) We vary $M$ by fixing $\alpha$. 2) We vary $\alpha$ by fixing $M$. Due to space limit, we defer the experiment results and analysis to the supplementary.

As shown in Fig. 7, we visualize the poll sampler’s scoring map, its sampled locations, and example aggregation weight map of the pool sampler. To summarize, 1) the poll sampler learns to sample the locations within and surrounding objects; 2) the pool sampler obtains different scales of context. For example, on the first row, the first pool sample attends to a wide range of spatial locations and encodes global context information; the second sample attends to a small area around the sky, and thus captures local context. We also have some other intriguing observations on the poll sampler: 1) It learns to sample object alike area beyond the object categories used for training. For example, for the last row in Fig. 7, locations around the traffic signs and the tree-like object are sampled. The behavior is similar to a learned region proposal network (RPN) [23], but learned without explicit supervision. 2) It tends to sample coarsely for some large and ‘easy’ objects but finely for small ones. For example, fewer points are sampled for the woman in the first row and the bed in the second row; the books in the second image and the cars in the last image are smaller and more difficult to detect, so the poll sampler finely samples feature vectors for those objects and surrounding areas.

To better understand the learning process and dynamics of the poll sampler, we record two statistics during training: (1) the proportion of sampled locations that are within the GT bounding boxes; (2) the pixel IOU of the sampled locations between consecutive epochs. As shown in Fig. 8, we make following observations. 1) The poll sampler gradually learns to sample more feature vectors that lie within the ground truth area but finally remains steady at about 60%, indicating that it also attends some background and contextual locations that are crucial for recognizing and detecting the objects. 2) The poll sampler initially has a large variation on its sampled locations, and thus the sampled areas of consecutive epochs have small IOU (i.e., about 0.2). During training, the IOU quickly converges to about 0.7 with around 30 epochs and remains steady at about 0.75, indicating that the sampler quickly learns to sample crucial feature vectors and the sampled locations does not change much. After learning rate decay at 100 epoch, the IOU of the consecutive epoch is close to 1.0, meaning the poll sampler converges.