Finding the Reflection Point: Unpadding Images to Remove Data Augmentation Artifacts in Large Open Source Image Datasets for Machine Learning

Data augmentation is a fundamental task in data preprocessing and training for deep learning tasks. However, when repurposing data between learning tasks or domains, images altered by task-specific augmentations are not always desired. Therefore, to recover the raw data when only an altered form is available, image restoration becomes a necessary computer vision subtask. Here, we introduce a novel and niche problem in image restoration, which can be introduced through errors in dataset curation during machine learning: the detection and removal of noisy mirrored regions. In mass image data collection or curation, especially when images may come from different cameras or feature cropped regions of interest of varying sizes, image dimensions may be inconsistent. However, many deep learning architectures require images of fixed input size. This is often remedied by resizing or padding; it is the padding case that we handle in this research.

As an alternative to padding at training time, machine learning practitioners may choose to pad their dataset prior to training as a precomputation step, which is also helpful in reducing repeated computations during multiple epochs of training or isolating features of interest for learning. Examples of such public dataset artifacts are shown in Figure 1. However, if this padded dataset is saved and publicized instead of the original images, the artifacts it contains can lead to problems in evaluation, especially when the data used is transferred to other tasks. With large and redundant data volumes, the padding may be fine for training, but during evaluation, the presence of padding – particularly symmetric or reflective padding – can simultaneously create realistic objects or patterns that should be recognized but are left out of annotation, and unrealistic objects or patterns that should not be recognized and actually distort the meaning of the original object. In both cases, for symmetrically-padded images, using annotations centered on object detection of the original objects will lead to misleading performance evaluation. A solution we propose in this research is image unpadding, where the padding on an image can be removed to restore an original image.

In cases of true zero-padding, the problem is trivial. When it comes to symmetric padding, further trivial methods, such as iterating through columns of pixels and identifying a consecutive repeating section in reverse, are only feasible when the image is saved without compression or resizing, as these processes can introduce interpolation noise. The presence of noise on either or both sides of a reflection makes detecting the boundary of an artificially-mirrored region non-trivial. Reflection removal requires a precise delineation of the mirrored boundary to separate authentic scene content from its redundant and often non-naturalistic counterpart.

In this research, we propose a systematic approach to identify the reflection boundary, accurately localize the mirrored region, and remove redundant, inaccurate, or non-naturalistic information. We then demonstrate the effectiveness of this unpadding on inference and evaluation of a zero-shot object detection task.

The necessity of image restoration, specifically unpadding mirrored regions, arises from the advent of image augmentation due to the necessity in variability and quantity of data with deep learning. To our knowledge, we are the first to address this issue and include further motivation in this section.

Having large volumes of high-quality data is paramount to the training of neural networks; for most tasks, the best-performing models are those that train on the greatest volume of data [29, 8, 17, 27]. In current tasks, this volume has increased to “internet-scale” amounts of images, which can be thought of (in some regard) as a ‘dataset augmentation’ by grabbing as much data as possible, even if from different sources, enabling transfer learning, foundation model learning, and more. This is a parallel path to standard data augmentation, where a dataset itself is used as the basis for generating different variations or augmentations on image samples. For example, a survey by Khalifa et al. [16] describes the benefits that data augmentation contributes to deep-learning models. They note that image augmentation overcomes data scarcity, with fields such as medical imaging lacking sufficient labeled data [3]. Additionally, compared to collecting and labeling new data, augmentation provides a cost-effective alternative by transforming existing data into new samples. Augmentation techniques help reduce overfitting and ensure models achieve higher accuracy during testing by diversifying training samples [28] and allowing for model generalizability [31, 23]. Finally, augmentation allows for more control over dataset characteristics, helping to balance class distributions and improve performance on imbalanced datasets [20].

Due to these advantages of both data and dataset augmentation, their usage is becoming prevalent. However, when altered datasets such as those exemplified in Figure 1 are released without the original naturalistic data, it is difficult for future researchers to repurpose and apply the dataset for their own training, evaluation, or dataset augmentation. Aligned with these concerns, these augmentation or padding artifacts can be detrimental to neural network training and validation [6], especially as augmented data does not accurately represent real-world data, further emphasizing the necessity of naturalistic original data.

Our algorithm for detecting the padded, mirrored area in an image is described in this section. We will explain the algorithm for just one side of the image (in this case, top), but we note that the algorithm should be applied to all four sides of the image.

We first create a variable for the dividing line, which iterates from the top to the middle of the image. At each position of the dividing line, we crop from the top of the image to the line as well as a section of the image with the same area as the crop right below the dividing line. We then mirror the first cropped image over the x-axis and obtain the mean squared error (MSE), calculated as

of these two cropped image segments. The line where the minimum MSE is located is then estimated to be the line between the padded area and the raw image, as the two cropped regions are most similar.

However, it is also noted that the given images do not always have mirrored padding. To detect these instances and ensure that we do not necessarily crop parts of the raw image, we determine a threshold for the MSE, where if the MSE is greater than the threshold, the obtained dividing line is disregarded. This is based on the assumption that images without reflected padding should have a higher MSE since the content of the cropped images across the dividing line is not identical.

When iterating the dividing line, we initially set it at some offset value from the boundary. This is since the portion of the image near the boundary sometimes has little to no difference in pixel values due to being part of the ground or sky. To clearly distinguish between unaltered images and padded images, the dividing-line offset ensures that MSE calculations focus on image regions where variability begins to appear. Without an offset, the minimum MSEs from unaltered images and padded images can be very low and overlapping in range due to cropping small sections of the image with low variability. By creating a clearer differentiation in the MSEs, we can better choose an MSE-threshold for unaltered and padded images. In the case of zero-padding, starting at the border would result in an extremely low MSE since the sections within the padding are identical. Therefore, it is necessary to start sufficiently positioned within the padding to increase the MSE. A pseudocode version of our algorithm is provided in Algorithm 1.

The threshold estimation method utilizing Otsu’s method is observed to have lower accuracy compared to iteratively testing the threshold as demonstrated in Table 1. This behavior is due to the MSEs and Otsu thresholds of the unpadded and padded images not having clear peaks and not being strongly separable. As shown in Figure 3, the MSEs of the unpadded images are dispersed, disallowing Otsu’s method to find a clear point for the threshold. Therefore, manually testing the threshold based on observed patterns in the MSEs would be more accurate. This causes the large discrepancy of the thresholds from the first and second methods.

Our threshold evaluation showed improved performance after removing padding, as it reduced ambiguities that could mislead the model. Additionally, the original dataset’s annotations for the mirrored padding were inconsistent, as often the entire person was not presented in the padding. This contributed to the model’s false negatives, contributing to an inaccurate representation of OWLv2’s performance. Figure 7 demonstrates both the missing ground truth annotations of the padded area as well as OWLv2’s imprecise detections in the symmetrically padded region. By removing the padding, our approach eliminates these ambiguous regions of missing annotations and distorted objects, leading to more accurate detections.

In this research, we proposed an algorithm to remove noisy artificially padded mirrored areas from images, utilizing a minimization of the MSE with potential borders of the original image. Our method has achieved robust performance, effectively identifying the transition between mirrored and non-mirrored regions. This led to a significant increase in performance on the task of zero-shot detection as we allow for more precise and reliable predictions and evaluations, with an increase from 0.47 to 0.61 in OWLv2’s average precision for hard hat detection and 0.68 to 0.73 in person detection.

Beyond artifact removal, ensuring that data augmentation techniques produce realistic transformations is an alternative solution to this reflective padding issue. Khalifa et al. [16] demonstrate this possibility as they describe that recent advances in augmentation now include complex strategies such as adversarial training, neural style transfer, and synthetic data generation rather than simple geometric transformations. By leveraging neural networks, they look to avoid augmentations that inaccurately represent real-world scenarios.

In light of recent trends in massive dataset augmentation by amalgamation of multi-source datasets to solve foundational learning tasks, we propose that the theme of finding unified visual qualities will be applicable in further use cases. For example, combined surveillance-related datasets may have combinations of both fisheye [10, 30] and non-fisheye lenses [18, 24], and detecting and correcting for these distortions is important for learning to detect visual patterns [32, 22]. The ability to recognize and correct for fundamentally disagreeing patterns between datasets will be important to rectify by restoring images before network training.

As large-scale machine learning continues to stay prevalent, ensuring high-quality, naturalistic data remains essential for enhancing generalization and real-world applicability across diverse applications.