Data Augmentation for Object Detection via
Differentiable Neural Rendering

Data Augmentation [16] is a powerful method to mitigate the problem of data scarcity, as augmented data will represent a potentially more comprehensive set of data points, closing the gap between the training and testing sets. Usually, the methods can be classified as data warping and oversampling. Data warping augmentations transform existing images while preserving the labels. Oversampling augmentations create synthetic instances to add into the training set, often used to re-sample imbalanced class distributions. Methods can also be categorized into online and offline depending on when the augmentation process occur.

In cognitive computer vision domain, most existing tasks are related to perception, i.e., perceive information from the image, video, voxel, mesh, or 3D point cloud. This is a process of either 2D or 3D reasoning. Typical tasks include object detection, human pose estimation, segmentation, 3D shape estimation, etc. In computer graphics domain, rendering is the process of generating images of 3D scenes defined by geometry, materials, scene lights and camera properties. The purpose of neural rendering [29, 30] is to bridge the gap between the 2D and 3D processing methods, allowing neural networks to optimize 3D entities while operating on 2D projections. Applications of differentiable neural rendering include: novel view synthesis [31, 32, 33, 34], semantic photo manipulation [35], facial and body reenactment [36, 37], re-lighting [38], free-viewpoint video [39], creating photo-realistic avatars [40] or simply generating high-quality images [41]. Our proposed augmentation method is inspired by novel-view synthesis [34], which is fully compatible with online data augmentation methods, and can be incorporated together to further inflate the dataset with novel semantics. GAN is described in [27] as a way to “unlock” additional information from a dataset. With neural rendering, we further unlock dataset information in highly controllable ways. The interpolation of data is non-linear and offers novel spatial semantics in 3D, which is extremely valuable for object detection tasks.

The overview of the DANR system is illustrated in Fig. 1. The purpose of the system is to augment an object detection dataset with novel view images to improve the performance of object detectors. The amount of augment images and degree of camera pose variations are both controllable. At its core, the system builds upon a novel view synthesis model, as shown in Fig. 2. The model takes as input an RGB image $I$ and a series of 2D image keypoints $B_{i}$ (representing bounding box annotations). The RGB image is simultaneously fed to two branches: (1) an hourglass network [42] for depth estimation, and (2) an Encoder with inception-resnet blocks [43] to extract visual features. The visual features are pixel-aligned, i.e., each pixel on the feature maps carries a latent-space vector that corresponds to this particular spatial location. The depth estimation branch assigns depth value to each pixel, elevating latent-space vectors to a point cloud. Each point in the cloud resides in a 3D space with the 2.5D coordinate representation [44]: $(u,v,d)$. Each point, however, may unfold its hidden N-dimensional latent-space vector after being projected to a target camera view. These feature vectors can be re-projected to an arbitrary camera coordinate system, based on a given projection matrix (translation vector $T$ and rotation vector $R$ for the corresponding camera of the original image). Given a target camera pose $P^{\prime}$, we splat the feature points following Wiles’s method [34] to make the rendering process differentiable. After splatted on a palette of size $K\times K$, these points result in an N-dimentional feature map of our chosen resolution. The feature maps are then fed to a Decoder network with inception-residual blocks and upsampling layers, rendering the final novel view image.

In order to recover the bounding box annotations for object detection datasets, we mark the keypoints of the bounding boxes beforehand. When they are projected to 3D space with depth information and re-projected back to 2D in a novel camera view, the marks consistently follow corresponding pixel-aligned points. The whole system is end-to-end differentiable.

In this section, we describe the data augmentation system in details. Firstly, we elaborate the state-of-the-art networks we employ and the specific settings. Then we introduce the point-cloud projection process and how to automate the generation of annotations in target views. Lastly, we depict the losses used to train the whole system.

This CCS is now regarded as the World Coordinate System (WCS). Given a specific target view, we can determine the extrinsic parameters. Transforming from the WCS to a novel CCS is straight-forward with $R$ and $T$:

where $P_{c}=(X_{c},Y_{c},Z_{c})$.

The points $P_{c}$ are then splatted to a target camera palette with new pixel locations:

Within each bounding box $i$, points (at least the four points at corners) are marked with an index value $i$. These marked keypoints are a subset of $S\times S$ points, where $S$ is the resolution of the extracted feature maps. When keypoints from the $i_{th}$ bounding box are re-projected to the target view after the procedures described above, their new pixel locations are denoted as $P^{\prime}_{i}=\{(u^{\prime}_{i},v^{\prime}_{i})\}$. The annotation for bounding box $i$ in the target image is computed as:

Keypoint estimation is naturally a regression problem [43]. The targets are represented by a series of heatmaps, each channel corresponding to a specific keypoint genre. Recent object detection methods such as CenterNet [2] , CornerNet [46] and ExtremeNet [47], begin to utilize keypoint estimation techniques. For instance, CenterNet transforms the task of object detection from generating and classifying proposals into predicting objects’ centers (keypoints) and corresponding attributes. To object detection, the attributes are the width, height of the object, along with local offsets that recover pixel location in the original resolution from down-sampled heatmaps.

We use CenterNet as our baseline detection framework to conduct experiments and ablation studies, due to its simplicity. Because it is anchor-free and does not need NMS as a post-processing step, the ablation study can be decoupled from complex design choices that is not concerned with training data. Anchor-free. During training, an object does not need to be assigned with proper anchor(s). Instead, only heatmaps for the entire image are generated as the regression target. NMS-free. When the heatmaps for object centers are inferred, the local peaks are ranked based on their response, and only top $K$ objects are extracted. With the center positions, corresponding attributes are extracted across their respective channels at the same 2D position. Since keypoints of different genres can occur at the same position, CenterNet is also naturally compatible with multi-label problems.

It is first demonstrated in [48] how mixing samples up could be developed into an effective augmentation strategy. Image mixup for the task of object detection has been described in [17], but it is restricted to bounding box mixup. We propose a straight-forward image mixup strategy for keypoint-based object detection methods, where ground truth keypoints are splat onto a heatmap ${Y\in[0,1]^{\frac{W}{R}\times\frac{H}{R}\times C}}$ using a Gaussian kernel:

$\sigma_{p}$ is an adaptive standard deviation [46] that is proportional to the size of the object. During mixup, the confidence of keypoint heatmaps are applied with the same weights used on its corresponding image, as shown in Fig. 3. This mixup strategy can be applied to keypoint-based object detection methods such as CenterNet. Compatible with the proposed offline augmentation method, image mixup is used in all of our experiments as an online augmentation method.

For DANR, we crop input images to $512\times 512$ for feature extraction and depth estimation. We also use a palette of the same size to splat points. We train on $4$ V100 GPUs with batch size $8$, randomly choosing samples for $500$ iterations per epoch. The sampled viewpoints of a reference video frame and its paired video frame are within $30$ frame range. The network is trained for a total of $275$ epochs, which takes $3.5$ days. Here are the corresponding metrics of the model, evaluated on RealEstate10K validation set: (1) the SSIM metric is $0.4386$; (2) PSNR is $7.1564$ and (3) perceptual score is $0.5964$.

We use CenterNet as our baseline detection framework to conduct experiments and ablation studies. We use identical training settings for all ablation experiments. Regardless of backbones, the network performs affine transform to keep aspect ratio and use $512\times 512$ image patch as input. Each network is trained for a total of $120$ epochs. We use Adam optimizer [57]. Learning rate is initiated to $2.5\times 10^{-4}$, which drops half twice at epoch $90$ and $110$, respectively. We do not perform any test-time augmentations, such as mirror flipping and multi-scale testing. For datasets except COCO, each image is augmented with four pre-defined views. They share the same translation vector $\begin{bmatrix}0&0&0.3\\ \end{bmatrix}$. Their respective rotation vectors are: $\begin{bmatrix}-0.1&-0.15&0\\ \end{bmatrix}$, $\begin{bmatrix}0.1&-0.15&0\\ \end{bmatrix}$, $\begin{bmatrix}-0.1&0.15&0\\ \end{bmatrix}$, $\begin{bmatrix}0.1&0.15&0\\ \end{bmatrix}$. For COCO dataset, we augment each image by randomly choosing one of the pre-defined views.

In this section, we ablate various choices in the use cases of our proposed DANR. For simplicity, all accuracy results here are for Indoor-Object validation set. Specifically, we show that the improvement is consistent regardless of detector configurations on various data distributions. We also discuss additional characteristics of augmentation such as the impact of image resolution. The following comparisons will be made. (1) Augmentation: online vs. offline; (2) render: high-resolution vs low-resolution; (3) Affine Transform: w/ affine vs. w/o affine; (4) data scarcity level: low vs. high; (5) detector: keypoints-based (one-stage) vs proposal-based (two-stage); (6) backbone: heavy vs. lightweight.

Table 2 shows that DANR is compatible with online mixup. Their augmentation strategies are complementary. DANR alone can boost the performance by around 10 percent.

In order to generate images at higher resolution, we experiment with 3 different settings: (1) Without any further training, we apply the network trained on a smaller image size to a different size at test time. Specifically, we splat the $256^{2}$ cloud points onto $512\times 512$ palette maps, then render the maps to an image with a refinement network. This augment method increased image resolution by super-sampling of splatted points, and is denoted as 512-pointSS. (2) We first augment images at $256\times 256$, and further upsample pixels at the image level with a pre-trained super-sampling network [58]. We denote this method as 512-pixelSS. (3) We re-train a network from scratch that takes in images at $512\times 512$, generate feature maps at $512\times 512$, and sample $512^{2}$ feature points in the cloud, which are splatted and rendered on the output image. This method naturally outputs images in $512\times 512$ resolution, denoted as 512-native.

As illustrated in Table 3, (1) augmentation with images rendered at different resolutions consistently boosts the detection performance; (2) synthesized images at low resolutions may potentially lose some details compared to real images, which does harm to the detection performance of very small objects; (3) uplifting the image resolution via super-sampling further improves the performance; (4) super-sampling from points may be a better choice than super-sampling from pixels; (5) training a network with dense features and points achieves the best performance. In the following experiments, we use 512-native as our resolution setting, as it achieves the highest AP and does not need any additional super-sampling networks.

Table 5 shows that the performance boost is closely related with the scarcity of training data. When annotated data is very limited, the DANR approach becomes more beneficial.

Table 6 shows that the proposed augmentation consistently boost the detection performance, regardless of the backbones used by the detector.

Summarizing the experimental results on various datasets, as shown in Table 8, we have come to the following insights: (1) Results on image content review datasets (ICR-Weapon, ICR-Flag and ICR-Logo) show that the proposed DANR is consistently helpful on real-world data when training data is scarce. (2) Results on indoor dataset show that the augmentation is highly beneficial with indoor data, even when training data is not scarce. (3) Results on COCO dataset have shown that DANR makes little contribution under circumstances where (a) the neural renderer in DANR fails to fully generalize to the semantics of the particular dataset, and (b) the training data is abundant.