Image-to-Image Translation-based Data Augmentation for Robust EV Charging Inlet Detection

Miseikis et al. used a shape-based template matching algorithm to localize charging ports in a pair of stereo images under a lab environment [2]. Depth is then estimated using corresponding coordinates on the image pair. Given 3D coordinates, perspective transformation is estimated using a least-squares fit algorithm to obtain refined position and orientation. Behl et al. presented a proof-of-concept robot to automatically charge an EV [1]. They divided the robot system into three subsystems and evaluated each subsystem independently. Regarding charging inlet plug-in/out, they used a 3D printed charging inlet instead of a real vehicle. They localized the inlet on a captured image from a camera by using the Hough transform [6] and a template matching algorithm. Long et al. proposed a design that uses both color camera(s) and range sensor(s) to achieve accurate localization [3]. Lou et al. proposed a cable-driven auto-charging robot that uses a vision sensor to localize charging ports [4]. However, the experiments were limited to a lab environment and used a simulated charging port. For both [3] and [4], localization algorithms were not specifically reported.

Another relevant work is charging inlet recognition. Sun et al. presented a CNN-based method that predicts whether an input image contains a complete, incomplete, fake charging port, or none [5]. However, since it does not include localization, it is not enough for automatic charging robots.

To the best of our knowledge, regarding charging inlet detection, previous works only conducted experiments using a simulated charging inlet under a lab environment. Also, the dataset is currently not publicly available. Hence, in this paper, we collected a novel dataset from real vehicles and will make it publicly available for further research.

Recently, day-to-night image translation has been studied for data augmentation [7, 8, 9, 10, 11]. Huang et al. proposed AugGAN that focuses on preserving objects on images during translation [7, 8]. They utilized the network to augment training data by day-to-night image translation to improve nighttime vehicle detection. Lin et al. extended AugGAN to Multimodal AugGAN to generate diversely translated images given an input image [9]. To achieve this, they utilized a low-dimensional latent vector to specify a condition for a translated image. Lee et al. presented a framework that uses CycleGAN [12] to translate daytime images to nighttime ones and trains a vehicle detector using both original and synthetic images [10]. Zheng et al. proposed ForkGAN that decouples domain-invariant content and domain-specific style information to translate nighttime images to daytime images or vice versa [11]. It was utilized to improve image localization/retrieval, semantic segmentation, and object detection.

While it was not utilized for data augmentation, HiDT network is also about day-to-night image translation [13]. Anokhin et al. presented the network to re-render an image to another one with different illuminations.

Another related work is translating nighttime images to daytime images to directly utilize the translated images for further applications [14, 11]. As the method aims to translate a test image to another one in an easier domain rather than augmenting training data, it does not require re-training final networks (e.g., object detection network). However, it requires translating images in a test phase, and accordingly increases computational demands during testing. Hence, it is less appropriate for real-time applications.

The proposed network is related to day-to-night image translation methods for data augmentation [7, 8, 9, 10, 11]. However, the proposed network utilizes an explainable environment vector to guide translated images. Also, by varying the vector, the network is able to generate various images given an image.

The methods for object detection can be divided into two main categories: one-stage methods and multi-stage methods. One-stage methods have advantages in being more efficient and simpler while multi-stage methods are more accurate and more flexible in general.

R-CNN is one of the earliest object detection methods that use CNNs [15]. Its improved versions were also proposed that reduce training and testing time [16, 17, 18]. They are all multi-stage methods that consist of region proposal step and detection step.

YOLO [19] and SSD [20] are two of the earliest one-stage methods. These methods employ a single neural network to predict class probabilities and bounding boxes directly from a whole image instead of each proposed region. As they do not have a separate region proposal step, they are usually faster than multi-stage methods. While they are faster, their accuracies are often lower. Lin et al. thought that this could be because of the extreme foreground-background class imbalance [21]. To overcome this, they proposed RetinaNet that utilizes focal loss. Tan et al. proposed EfficientDet [22] that uses a compound scaling method similar to EfficientNet [23] to generate a family of one-stage object detectors.

DetectoRS [24] proposed the recursive feature pyramid and applied it to a multi-stage detector, HTC [25]. The recursive pyramid adds an additional feedback connection from the top-down path to the bottom-up path in the feature pyramid network. It also utilizes switchable atrous convolution that outputs the weighted summation of the outputs from two convolution paths with different atrous rates.

Recently, methods that utilize Transformer [26] were also proposed [27, 28, 29]. Carion et al. proposed DETR that first extracts features using CNNs and employs transformer encoder/decoder [27]. Given object queries, the decoder outputs the corresponding embeddings based on the encoder’s output in parallel. The embeddings are then processed by a shared feed-forward network to obtain detection results. To reduce computational demands of DETR, Zhu et al. proposed Deformable DETR [28], and Zheng et al. proposed Adaptive Clustering Transformer [29].

As we propose a novel detection dataset, we experimented with various object detection networks [21, 17, 18, 24, 27] and report their performances.

We collected a new dataset using Intel RealSense Depth Camera D435i since we were not able to find a publicly available dataset for EV inlet detection. The collected dataset consists of 4,153 pairs of color images and depth maps. Depth maps were only used to get more robust ground truth annotations, especially for images collected at night. The dataset was collected from two vehicles at various environments (locations, times, and others). The two vehicles are Chevrolet Bolt EV 2019 and Kia Niro EV 2018. Ground truth bounding boxes were labeled by human annotators. Specifically, three components of a charging inlet were annotated so that not only location but also rotation can also be estimated (see Fig. 3). Example images are shown in Fig. 4, and detailed information is provided in Table I.

Among 4,153 images, 1,009 and 1,194 images were collected at daytime/morning and at night/evening, respectively. The remaining 1,950 images were collected at indoor parking facilities at various times. The dataset is used to create two benchmark datasets (EVCI-A and EVCI-B dataset). EVCI-A dataset is designed to train, validate, and test on all environments (indoor, daytime/morning, and night/evening) (see Table II). For the EVCI-B dataset, the training dataset consists of images collected in daytime/morning and at indoor facilities while the validation/testing datasets consist of images captured at night/evening and indoor facilities mostly. Hence, by utilizing the EVCI-B dataset, given only images from daytime/morning/indoor for training, performances on images from night/evening can be measured.

In general, contrast, brightness, and saturation vary depending on whether an image is captured under daytime lighting levels or night-time lighting [30]. Since our goal is, given an image, generating various images that look like images captured under different lighting conditions, we propose to utilize brightness, contrast, and saturation to guide an environment. So, given an image ($\boldsymbol{I}$) and an environment guide vector ($\boldsymbol{e}\in\mathcal{R}^{3}$), a corresponding image is generated.

To validate the effectiveness of the proposed environment guide vector, we analyzed the brightness, contrast, and saturation of the images in the EVCI dataset. Specifically, cumulative distribution functions of each of them are computed for images captured under indoor, daytime, morning, night, evening environments (see Fig. 5). We also show the 3D scatter plot of data where each axis corresponds to each of them in Fig. 6. The left and right plots are the same except for their axes. We can observe that most night-time images have low brightness and low contrast. We can also observe that most indoor images have low saturation while most evening images have high saturation. Please note that the distribution might be different from natural images as the captured environments might have lighting from vehicles, street lamps, or any artificial lighting (see Fig. 3 and Fig. 4).

In more detail, brightness ($\boldsymbol{e}^{\prime}_{1}$) is computed by the mean of pixel-wise luminance, which is calculated by the method in [31]. Root mean square (RMS) contrast is used for contrast ($\boldsymbol{e}^{\prime}_{2}$), which is the standard deviation of pixel-wise luminance. Saturation ($\boldsymbol{e}^{\prime}_{3}$) is computed by the mean of pixel-wise saturation, which is calculated by the same way as the HSV color representation. The environment vector $\boldsymbol{e}$ is then determined by normalizing each component of the computed vector $\boldsymbol{e}^{\prime}$ to $[-1,1]$. The normalization is processed using the minimum and maximum of each component of all images in the dataset.

The proposed image-to-image translation network is designed to be able to synthesize an image that a human can easily control and explain. Also, it is to generate various images given an image. The training framework of the proposed network is shown in Fig. 7. $E_{c}$ and $E_{env}$ represent the content encoder and the environment guide vector extractor, respectively. $E_{c}$ aims to encode content information and consists of three convolution layers and five residual blocks where each residual block contains two convolution layers and one skip connection. $E_{env}$ aims to extract environment information such as lighting conditions, the existence of artificial lighting, etc. It is designed to be able to intuitively control synthesized images. It extracts brightness, contrast, and saturation as explained in Section III-B and is a non-learnable component. $G$ represents the generator that synthesizes an image given an encoded content map $\boldsymbol{c}$ and an environment guide vector $\boldsymbol{e}\in\mathcal{R}^{3}$. The environment vector is first processed by an MLP that consists of three fully connected layers. Then, adaptive instance normalization (AdaIN) [32] is applied to the encoded content map using the MLP-processed environment vector. Finally, the output of the AdaIN layer is processed by four residual blocks, two transposed convolution layers, and one convolution layer to obtain synthesized images.

In the inference stage, given an image and an environment vector, the generator $G$ synthesizes an image along with the content encoder $E_{c}$. The environment vector can either be human selected, randomly generated, or extracted from an image (using environment vector extractor $E_{env}$).

In the training phase, the network is trained using the following objective functions:

where $\mathcal{L}_{E,G}$ denotes the loss that is used to train the generator and the content encoder. The loss consists of image reconstruction loss, cycle consistency loss, environment translation loss, perceptual loss, and adversarial loss for generators. $\mathcal{L}_{D}$ contains the adversarial loss for discriminators.

Image reconstruction loss. The loss $\mathcal{L}_{rec}$ is to be able to reconstruct an input image if its encoded content map $\boldsymbol{c}$ and its extracted environment vector $\boldsymbol{e}$ are given to the generator $G$ as inputs. The loss is defined as the $\ell^{1}$-norm of the pixel-wise differences between the input image $\boldsymbol{I}$ and the reconstructed image $\hat{\boldsymbol{I}}$. Formally, it is defined as $\mathcal{L}_{rec}=\lVert\hat{\boldsymbol{I}}-\boldsymbol{I}\rVert_{1}=\lVert G(\boldsymbol{c},\boldsymbol{e})-\boldsymbol{I}\rVert_{1}=\lVert G(E_{c}(\boldsymbol{I}),E_{env}(\boldsymbol{I}))-\boldsymbol{I}\rVert_{1}$.

Cycle consistency loss. The loss $\mathcal{L}_{cyc}$ has similarity with the image reconstruction loss as it is also about reconstructing an input image. The difference is that the reconstructed image $\bar{\boldsymbol{I}}$ is generated given the encoded content map $\tilde{\boldsymbol{c}}$ of another synthesized image $\tilde{\boldsymbol{I}}$ along with the input image’s environment vector $\boldsymbol{e}$. $\tilde{\boldsymbol{I}}$ is generated using the encoded content map $\boldsymbol{c}$ of the target input image $\boldsymbol{I}$ and an environment vector $\boldsymbol{e}^{\prime}$ from another image $\boldsymbol{I}^{\prime}$. It is defined as follows:

Environment translation loss. The loss $\mathcal{L}_{env}$ is to ensure that a generated image has the characteristics that are guided by the given environment vector. It measures the differences between the given environment guide vector to a generator and the extracted environment vector from the synthesized image from the generator. Considering Fig. 7, as three generators exist, three corresponding environment translation losses are computed. The total environment translation loss $\mathcal{L}_{env}$ is the summation of them ($\hat{\mathcal{L}}_{env}$, $\tilde{\mathcal{L}}_{env}$, $\bar{\mathcal{L}}_{env}$).

Perceptual loss. The loss $\mathcal{L}_{perc}$ is to enforce the consistency of contents between an input image and the generated image using the encoded content map of the input image. To achieve this, the loss measures the difference between the encoded content map $\boldsymbol{c}$ of the input image and that $\tilde{\boldsymbol{c}}$ of the synthesized image $\tilde{\boldsymbol{I}}$ where the latter image is generated using $\boldsymbol{c}$ and another image’s environment vector $\boldsymbol{e}^{\prime}$. The loss is defined as follows:

Adversarial loss. The loss $\mathcal{L}_{adv}$ is employed to ensure that the distribution of generated images is similar to that of real images in the dataset. Similar to [11, 12, 13], we utilize the least squares loss from LSGAN [33]. We also employ multi-scale discriminators where we used two scales. One discriminator is applied at generated images’ resolution, and the other takes images at half resolution. It is defined as follows:

where $S(\cdot)$ denotes downsampling by 2.

Final loss. The generator and the content encoder are trained by minimizing $\mathcal{L}_{E,G}$ that is the weighted summation of $\mathcal{L}_{rec}$, $\mathcal{L}_{cyc}$, $\mathcal{L}_{env}$, $\mathcal{L}_{perc}$, and $\mathcal{L}_{adv_{G}}$. The discriminator is trained by minimizing $\mathcal{L}_{D}$.

The network is trained using the Adam optimizer for 20 epochs with the initial learning rate of 0.0002. The learning rate was linearly decayed after 10 epochs. We used the following hyperparameters: $\lambda_{1}=\lambda_{2}=10/hw$, $\lambda_{3}=0.5$, $\lambda_{4}=1/h^{\prime}w^{\prime}$, $\lambda_{5}=\lambda_{6}=0.5$ where $(h,w)$ and $(h^{\prime},w^{\prime})$ denote the height and width of an input image and the corresponding content map, respectively. In the training phase, the input image is resized to $286\times 572$ and then is scaled by a factor that is randomly sampled from the uniform distribution $[0.9,1.1]$. It is then randomly cropped to be $256\times 512$ and is processed by random horizontal flipping.

To demonstrate that the data augmentation using the proposed network improves a state-of-the-art detection network, we first analyzed multiple detection networks [17, 18, 21, 24, 27] (see Table III). Among them, the highest accuracy is achieved by DetectoRS [24]. Hence, we compare the performances of it trained using the original training dataset, using the ForkGAN-augmented dataset [11], and using the proposed EnT-GAN-augmented dataset.

For both ForkGAN [11] and the proposed EnT-GAN, a detection network is trained using augmented training data that consists of both original training images and translated images. The distribution of data during training is 50% from original training images and the remaining 50% from translated images. For the proposed EnT-GAN, environment vectors are randomly generated from the uniform distribution $[-1,1]$ unless specifically mentioned. Also, since EnT-GAN can generate various images given an image, varying sets of translated images are used during training.

The detection network is trained using stochastic gradient descent (SGD) optimizer for 30 epochs with linearly increasing learning rate until 500 iterations to 0.002/0.001 for the EVCI-A/B dataset. The input image is first resized to $800\times 1333$ while keeping aspect ratio and is randomly flipped horizontally. We also show the result of utilizing random scaling/cropping. For the random scaling, a scaling factor is sampled from the uniform distribution $[0.9,1.1]$. The scaled image is then randomly cropped to be $384\times 600$.