Decoupled Dynamic Filter Networks

Convolution is a fundamental building block of convolutional neural networks (CNNs) that have seen tremendous success in several computer vision tasks, such as image classification, semantic segmentation, pose estimation, to name a few. Thanks to its simple formulation and optimized implementations, convolution has become a de facto standard to propagate and integrate features across image pixels. In this work, we aim to alleviate two of its main shortcomings: Content-agnostic and Computation-heavy.

In this work, we propose the Decoupled Dynamic Filter (DDF) that simultaneously addresses both the above-mentioned shortcomings of the standard convolution layer. The full dynamic filter [57, 23, 49, 45] uses a separate network branch to predict a complete convolution filter at each pixel. See Figure 1 (middle) for an illustration. We observe that this dynamic filtering is equivalent to applying attention on unfolded input features, as illustrated in Figure 3. Inspired by the recent advances in attention mechanisms that apply spatial and channel-wise attention [36, 50], we propose a new variant of the dynamic filter where we decouple spatial and channel filters. In particular, we adopt separate attention-style branches that individually predict spatial and channel dynamic filters, which are then combined to form a filter at each pixel. See Figure 1 (bottom) for an illustration of DDF. We observe that this decoupling of the dynamic filter is efficient yet effective, making DDF to have similar computational costs as depth-wise convolution while achieving better performance against existing dynamic filters. This lightweight nature enables DDF to be directly inserted as a replacement of the standard convolution layer. Unlike several existing dynamic filtering layers, we can replace all $k\times k$ $(k>1)$ convolutions in a CNN with DDF. We also propose a variant of DDF, called DDF-Up, that can be used as a specialized upsampling or joint-upsampling layer.

We empirically validate the performance of DDF by drop-in replacing convolution layers in several classification networks with DDF. Experiments indicate that applying DDF consistently boosts the performance while reducing computational costs. In addition, we also demonstrate the superior upsampling performance of DDF-Up in object detection and joint upsampling networks. In summary, DDF and DDF-Up have the following favorable properties:

• •

  Content-adaptive. DDF provides spatially-varying filtering that makes filters adaptive to image contents.

• •

  Fast runtime. DDF has similar computational costs as depth-wise convolution, so its inference speed is faster than both standard convolution and dynamic filters.

• •

  Smaller memory footprint. DDF significantly reduces memory consumption of dynamic filters, making it possible to replace all standard convolution layers with DDF.

• •

  Consistent performance improvements. Replacing a standard convolution with DDF / DDF-Up results in consistent improvements and achieves the state-of-the-art performance across various networks and tasks.

The goal of this work is to design a filtering operation that is content-adaptive while being lighter-weight than a standard convolution. Realizing both the properties with a single filter is quite challenging. We accomplish this with our Decoupled Dynamic Filter (DDF), where the key technique is to decouple dynamic filters into spatial and channel ones. More formally, the DDF operation can be written as:

where ${F^{\prime}}_{(r,i)}\in\mathbb{R}$ denotes the output feature value at the $i^{th}$ pixel and $r^{th}$ channel, ${F}_{(r,j)}\in\mathbb{R}$ denotes the input feature value at the $j^{th}$ pixel and $r^{th}$ channel. $D^{sp}\in\mathbb{R}^{n\times k\times k}$ is the spatial dynamic filter with $D_{i}^{sp}\in\mathbb{R}^{k\times k}$ denoting the filter at $i^{th}$ pixel. $D^{ch}\in\mathbb{R}^{c\times k\times k}$ is the channel dynamic filter with $D_{r}^{ch}\in\mathbb{R}^{k\times k}$ denoting the filter at $r^{th}$ channel. Figure 2(a) shows the illustration of DDF operation. We predict both channel and spatial dynamic filters from the input feature, using which we perform the above DDF operation (Eq. 2) to compute the output feature map. Comparing general dynamic filters (See Section 3) with DDF clearly indicates that DDF reduces the $nc^{\prime}ck^{2}$ sized dynamic filter into much smaller $nk^{2}$ spatial and $ck^{2}$ channel dynamic filters. In addition, we implement DDF operation in CUDA alleviating any need to save intermediate multiplied filters during network training and inference.

Following the recent advances in attention literature [36, 50] that propose to use lightweight branches to predict spatial and channel-wise attention, we design two attention-style branches that can generate spatial and channel dynamic filters for DDF. Figure 2(b) illustrates the structure of spatial and channel filter branches in the DDF module. The spatial filter branch only contains one $1\times 1$ convolution layer. The channel filter branch first applies the global average pooling to aggregate input features, then generates channel dynamic filters via a squeeze-and-excitation structure [18], where the squeeze ratio is denoted as $\sigma\in\mathbb{R}^{+}$.

As generated filter values can be extremely large or small for some input features, directly using them for convolution will make the training unstable. So, we propose to do filter normalization (FN):

where $\hat{D}^{sh}_{i},\hat{D}^{ch}_{r}\in\mathbb{R}^{k\times k}$ are the generated spatial and channel filters before normalization, $\mu(\cdot)$ and $\delta(\cdot)$ calculate the mean and standard deviation of the filter, $\alpha^{sp},\alpha^{ch}_{r},\beta^{sp},\beta^{ch}_{r}$ are the running standard deviation and mean values which are similar to those coefficients in the batch normalization (BN) [20]. FN can limit generated filter values into a reasonable range, thereby avoiding the gradient vanishing/exploding during training.

Image classification is considered as a fundamental task in computer vision. To demonstrate the use of DDFs as basic building blocks in a CNN, we experiment with the widely used ResNet [15] architecture for image classification. ResNets stack multiple basic/bottleneck blocks in which $3\times 3$ convolution layers are adopted for spatial embedding. We substitute these $3\times 3$ convolution layers in all stacked blocks with DDF. We refer to such a modified ResNet with DDF as ‘DDF-ResNet’. Figure 4 illustrates the use of DDF in a ResNet bottleneck block, we refer to it as DDF bottleneck block.

We evaluate DDF-ResNets on the ImageNet dataset [10] with the Top-1 and Top-5 accuracy. DDF-ResNets are trained using the same training protocol as [27]. In particular, we train models for 120 epochs by the SGD optimizer with the momentum of 0.9 and the weight decay of 1e-4. The learning rate is set to 0.1 with batch size 256 and decays to 1e-5 following the cosine schedule. The input image is resized and center-cropped to $224\times 224$.

By replacing the convolution with the channel dynamic filters, the top-1 accuracy is improved by 1.6%. Using the full DDF module, with both spatial and channel dynamic filters, improves the top-1 accuracy by 1.9%. These results show the importance of both the spatial and channel dynamic filters in DDF.

Table 4(b) compares different normalization schemes in a DDF module. Replacing the proposed filter normalization with a standard batch normalization [20] or a sigmoid activation leads to considerable drops in accuracy. Sigmoid activation individually processes each filter value and may not capture the correlation between them, while batch normalization considers all the filters in a batch, which may weaken the filter dynamics across samples.

We also evaluate DDF under different squeeze ratios $\sigma$, which is used to control the feature channel compression in the channel filter branch. As shown in table 4(c), using higher squeeze ratios will significantly increase the number of parameters, while only bringing marginal performance improvements. Hence, we set the squeeze ratio to 0.2 by default. In addition, even the parameter number increases with enlarging the squeeze ratio, the FLOPs remain low because the computational costs of the channel filter branch are minimal, as analyzed in Section 4.1.

(more analysis in the supplementary).

Recently, neural architecture search (NAS) methods [44, 33] can obtain architectures with outstanding speed/accuracy trade-off. The proposed DDF module can also contribute to the search space of NAS methods as a new fundamental building block.

An advantage of dynamic filters compared with standard convolution is that one could predict dynamic filters from guidance features instead of input features. Following this, we propose an extension of the DDF module, where spatial dynamic filters are predicted using separate guidance features instead of input features. Such joint filtering with input and guidance features is useful for several tasks such as joint image upsampling [42, 28, 29], cross-modal image enhancement [51, 9, 6], texture removal [32] to name a few. Figure 5 illustrates the DDF Upsampling (DDF-Up) module, where the number of DDF operations used is set to $x^{2}$ for the upsampling factor $x$ (e.g., 4 DDF operations when the upsampling factor is 2). We stack and pixel-shuffle [40] the resulting features from the DDF operations to form output features. For typical upsampling (without guidance), we use the same structure with a slight modification. We compute guidance features from input ones using a depth-wise convolution layer. DDF-Up can be seamlessly integrated into several existing CNNs, where typical/joint upsampling operators are needed. Here we present two applications in object detection and joint depth upsampling tasks.

We analyze the effectiveness of DDF-Up modules with experiments on the COCO detection benchmark [31] which contains 115K training and 5K validation images. We report standard COCO [31] metrics for small (mAp_S), medium (mAp_M), large (mAp_L), and all-scale (mAp) objects on the minival split. We implement our models based on the MMDetection [3] toolbox and train them using the standard training protocol therein. Specifically, we train different models for 12 epochs using the SGD optimizer with a momentum of 0.1 and the weight decay of 1e-4. We use a batch size of 16 and set the learning rate to 0.2 which decays by the factor of 0.1 at 8 and 11^th epochs. We resize the shorter side of the input image to 800 pixels, while keeping the longer side no larger than 1333 pixels.

We compare DDF-Up with the generic nearest-neighbor (Nearest) and bilinear (Bilinear) interpolations, as well as learnable Deconvolution (Deconv) [35], Pixel Shuffle (P.S.) [40], and CARAFE [48] upsampling modules. Table 6 exhibits the comparison results. FPN with DDF-Up yields a 1.2% mAp improvement over the baseline which adopts the nearest-neighbor interpolation. DDF-Up also brings obvious improvements against static-filtering upsampler (like Deconv and P.S.), and is on par with the recent dynamic upsampling technique CARAFE while utilizing only one-third of FLOPs as CARAFE.

We conduct experiments on the NYU depth V2 dataset [41] which has 1449 RGB-depth pairs. Following PAC-Net [42], we use the nearest-neighbor downsampling to generate low-resolution inputs from the ground-truth (GT) depth maps. We split the first 1000 samples for training and the rest for testing. We train DDF-Up-Net for 1500 epochs using the Adam optimizer [24]. We use a batch size of 8 and set the learning rate to 1e-4 which decays by the factor 0.1 at 1000 and 1350^th epochs. During training, the input images are resized and random-cropped to $256\times 256$. Table 7 reports Root Mean Square Error (RMSE) scores of different techniques for three upsampling factors, \ie, $4\times$, $8\times$, and $16\times$. DDF-Up-Net performs better than state-of-the-art techniques across all the upsampling factors. It surpasses the standard CNN techniques like DJF [28] and DJF+ [29] by a large margin. It also improves over dynamic-filtering PAC [42] while reducing computational costs by an order of magnitude. See Table 2 for the cost comparison between PAC and DDF-Up. We visualize sampled 16$\times$ upsampling results in Figure 7, where we can see that DDF-Up-Net recovers more details compared to PAC-Net and other techniques.

In this work, we propose a lightweight content-adaptive filtering technique called DDF, where our key strategy is to predict decoupled spatial and channel dynamic filters. We show that DDF can seamlessly replace standard convolution layers, consistently improving the performance of ResNets while also reducing model parameters and computational costs. In addition, we propose an upsampling variant called DDF-Up, which boosts performance as both a general upsampling module in detection and a joint upsampling module in joint depth upsampling. DDF-Up also is more computationally efficient compared with specialized content-adaptive layers. Overall, DDF has rich representative capabilities as a content-adaptive filter while also being computationally cheaper than a standard convolution, making it highly practical to use in modern CNNs.

This work is supported in part by the National Natural Science Foundation of China (No.61976094). M.-H. Yang is supported in part by NSF CAREER 1149783.