Towards Accurate and Robust Classification in Continuously Transitioning Industrial Sprays with Mixup

Despite its detrimental effects on ICE performances, spray collapse or spray morphology classification remains a difficult problem. This challenge is further exacerbated from design changes such as the move from outward opening swirl injectors to multi-hole direct injection injectors. With recent advances in turbo-charging, downsizing, and new injection strategies, domain experts are increasingly relying on experiments or high-fidelity simulations to predict spray structures, especially the transition to spray collapse. Many of the reported work focus on developing experimental correlations based on carefully designed experiments. However, due to the limitation of hardware and resources, the reported correlations rarely give satisfactory results on conditions out of the original research scope. Some of the works even reported contradictory findings. Zeng et al. [8] found that the spray morphology transition can be solely predicted by the ratio of ambient pressure to fuel saturation vapor pressure. The tested spray stays in the transitional regime when the pressure ratio is between 0.3 to 1.0, and spray collapse happened when its value is less than 0.3. On the contrary, Lacey et al. [9] reported that the pressure ratio was not effective at predicting the transition to spray collapse, especially of different fuels. Various theories [10, 11, 12] have been proposed in the literature, but a universally applicable model has yet to be discovered.

It should be noted that we do not criticize the findings in  [8, 9, 10, 11, 12]. In fact, we believe each model or theory is perfectly explainable to the researchers within the scope of their studies. However, each study represents an enormous effort in the design and conduct of the experimental measurements, which inevitably, resulted in a limited dataset for analysis. Therefore, those findings are limited in the sense that they can only provide accurate information within the data space explored by each corresponding study.

Though deep neural networks have found various applications in the scientific domain such as medical imaging, materials discovery, computational fluid dynamics to name a few. To the authors’ best knowledge there are no efforts in the classification of continuously transitioning spray structures. On the training front new methods and/or training techniques have been reported since the original work of Mixup by Zhang et al. [6], which demonstrated that Mixup reduces test errors of multiple state-of-the-art deep convolutional neural network models on ImageNet, CIFAR, and speech data. Berthelot et al. [13] expanded the idea to semi-supervised learning (SSL) and proposed a ”holistic” learning method named MixMatch. Their experiments on SSL suggest that MixMatch significantly improved performance compared to other methods they studied. The same group took a step further and proposed the ReMixMatch by introducing augmentation anchoring and distribution alignment to MixMatch [14]. Their experiments show that this SSL algorithm can reach or beat MixMatch with much less data. Jeong et al. [15] used Mixup as one of the functioning elements and proposed an Interpolation-based Semi-supervised learning method for object detection (ISD). They demonstrated that ISD significantly improves the performance of Single Shot Multibox Detector(SSD) [16] in both supervised and semi-supervised object detection tasks.

Due to the nature of our problem, the entire dataset is composed of proprietary spray images collected from our internal spray laboratory. We wish to develop a deep learning model that is capable of classifying spray structures over the various testing conditions, injection timings, and experimental setups. This allowed spray images to be collected from multiple sources, covering a wide range of operating conditions, injector geometries, and measurement events. Another consideration during the data collection process is that we wish to cover the entire injection event, so an additional class named ”Pre/Post” (short for pre-injection and post-injection) was added to the dataset so the final model can take any input from the Mie Scattering without human attention.

Table 1 lists the number of examples for each class, giving a total of 878 images. We set aside 75% of the dataset for training, 15% for validation, and 10% for testing, leaving a total of 658, 152, and 88 images for training, validation, and testing, respectively. Examples of spray images in each class are shown in Figure 2. Since the data were collected from multiple sources, they are not necessarily the same size as can be seen in Figure 2. In addition, some of the images may carry auxiliary boundary lines and/or text annotations processed by Mie Scattering tool. Those lines were not removed during the labeling process and we anticipate the model would either treat them as noise or recognize them as useful features.

Given the limited dataset, the most feasible starting point is transfer learning by reusing the lower layers of pre-trained models. Fortunately, many state-of-the-art deep learning models are released for Keras [17]. TensorFlow version 2.4.1 [18] is used throughout the work. Among the many available pre-trained models on Keras, we chose ResNet [19] family as the starting point after an internal evaluation. All the images were pre-processed as $224\times 224$ pixel images as expected by ResNet. In addition, the grayscale images were loaded in Red, Green, and Blue channels. As shown in Figure 3, loading grayscale images in RGB channels leads to three nearly identical colored images. These were then fed into pre-trained ResNet models and fine tuned.

To select a suitable model for fine tuning, we performed 5-fold cross validation using all 878 available spray images. Seven variants of ResNet models were tested. Note that we implemented ResNet34 from scratch since it is not available from Keras. For the remaining six models, we reused all the pre-trained lower and mid layers and only made the last block (namely, conv5 block3) trainable. This leaves about 4.4 million trainable parameters for each model except for ResNet34. We also removed the fully connected top layer and replaced it with global average pooling layer, followed by a dense output layer with one neuron per class. For pre-trained ResNet models, we used Adam optimizer [20] and fixed the learning rate, batch size, and total epochs to be 0.001, 32, and 100, respectively. For ResNet34, we used stochastic gradient descent with a learning rate of 0.0001 and momentum of 0.9.

The ensemble-averaged test accuracies and standard deviations are reported on Table 2. ResNet50 outperforms all other models with an impressive test accuracy of 96%, though other models are not far behind. Although ResNet101 and larger models have more representation power than ResNet50, they are prone to overfitting with small training datasets and therefore have slightly poor performance on the test set. On the other hand, ResNet34 seems too shallow to learn all necessary low-level and/or high-level features. Given its highest accuracy, ResNet50 is used as the base model for further experiments.

Mixup is a simple and data-agnostic method proposed by Zhang et al. [6]. Suppose we have two input image vectors, $x_{1}$ and $x_{2}$, and their corresponding one-hot vectors are $y_{1}$ and $y_{2}$, then the Mixup augmented training image vectors are given by:

where $\lambda$ is the interpolation coefficient randomly drawn from the Beta distribution, $\lambda\sim\mathbf{Beta}(\alpha,\alpha)$. As shown in Figure 4, $\lambda$ approaches 0 or 1.0 as $\alpha\rightarrow 0$, this essentially eliminates interpolation and selects one input image as the output. As $\alpha$ increases, a realization of $\lambda$ would have a higher chance of being close to 0.5, leading to strong blending of both input images.

There are many choices of Mixup implementations as suggested by Zhang et al. [6]. For example, the augmented one-hot vector can be the same as the one input image with larger weight. Mixup can also be performed on more than two images. In this work, we implemented the vanilla version given by Equation 1.

Figure 5 shows some examples after Mixup with different $\alpha$ values. Note that we only tested $\alpha$ up to 0.6 as Zhang et al. [6] found Mixup only improved performance over traditional data augmentation with $\alpha<0.4$. For larger $\alpha$ values, Mixup leads to underfitting. At $\alpha=0.2$, four out of 20 randomly selected images are visually discernible as being augmented by Mixup. Among those four images, one of them was interpolated between two images belongs to the same class as highlighted by green cycle. The resulting image expands the data distribution of that class and hence Mixup works like a data augmentation tool. The other three were interpolated between two different classes, the resulting labels are no longer one-hot vectors. This is considered crucial to our application as those cross-class augmented images mimic the smooth transition observed in our dataset. As $\alpha$ increases, the interpolation becomes stronger, and more images can be observed as being interpolated.

Data augmentation is widely used to combat overfitting of deep neural networks. However, among many choices of data augmentation methods such as shifting, rotation, flip, zooming, one must be careful as the effectiveness of data augmentation is dataset dependent and use of domain knowledge is usually needed. Although Mixup was found to improve performance over data augmentation [6], they can easily work together. In this work, Mixup is performed before any optional data augmentation since it guarantees that there will be only one injector tip located on the top of each augmented spray image. If image augmentation such as rotation and shifting are applied before Mixup, then the Mixup augmented image may have two injector tips, which from the physics point of view pollutes the dataset and will have negative effects on the model performance.

At this stage, we introduce some data augmentations without Mixup to fine-tune the ResNet50 model. Like section 4.2, only the last block of ResNet50 (i.e., conv5 block3) is allowed to be retrained on our dataset. The lower- and mid-level layers from Keras ResNet50 model pre-trained on ImageNet are reused. Random rotation up to 20 degrees is used since in our experiments, the injector may be installed at a slightly angled position. Random shifting is limited up to 20% as the spray image is almost always centered at the camera window. Horizontal flip is allowed as it produces an image taken at the opposite direction. Vertical flip is not used since it gives an upside-down image that makes no physical sense. Brightness, saturation, hue, or contrast adjustments are excluded as well given Mie Scattering imaging produces consistent images without much distortion. We use Adam optimizer [20] with an initial learning rate of 0.001, and it is then divided by 10 after no improvements observed in validation loss for 5 consecutive epochs. We also use early stopping with a patience of 50 to avoid over-training. In all our experiments, the model will stop training with less than 200 epochs.

The ensemble-averaged training-validation accuracy gaps over the last 50 epochs are reported on Table 3, also reported are the test accuracies of the final model. Compared to the baseline without any data augmentation, all single data augmentations lead to improved generalization. Combining two or three augmentation methods do not bring the training-validation accuracies any closer. Another observation is that the test accuracy saturates with all experiments. Given we only have 88 test images, 94%, 96.5%, and 97.7% test accuracies indicate there are only 5, 3, and 2 images being miss-classified by the model. Note that those miss-classified images are not necessarily the same between each test. Although there is no universal guideline as how small the training-validation gap should be to indicate a good model, we argue the value should be at least less than 1.0% for this application.

We evaluate Mixup without data augmentation and summarize the results on Table 4. Note that all Mixup tests lead to improved performance over data augmentation. The training-validation accuracy gaps are below 1.0% with all three $\alpha$ values as we desired. Larger $\alpha$ leads to better generalization, but the model under-fits the training set as evidenced by the gradually decreasing ensemble-averaged training accuracies. This finding is consistent with the work by Zhang et al. [6], although they use much bigger datasets (ImageNet).

Zhang et al. [6] introduced ”corrupted labels” to CIFAR dataset by replacing up to 80% of total image labels with random noises. Their testing showed that Mixup can mitigate the memorization of those corrupted labels. In our application, some ”corrupted labels” are unintentionally introduced because of the blurry boundaries between continuous class transitions. We hypothesize that Mixup can help combat the memorization of those ”corrupted labels” as well. For example, if one image within the ”collapse/transitional” is labeled as ”collapse” and its neighbor from the next camera frame is labeled as ”transitional”, then a neural network without Mixup would learn to memorize those labels and over-fit the data. On the other hand, all training images are generated from Mixup (although not all of them are interpolated), so those two images may produce a new training image with an interpolated label. This is equivalent to labeling the newly generated image as ”collapse/transitional”, thereby reduces the memorization of ”corrupted labels”. From the physics point of view, this interpolation also aligns with the actual transition in the experiments.

We also perform a test with Mixup ($\alpha=0.2$) followed by the ”best-practice” data augmentation reported on Table 3, i.e., shifting by 20% plus horizontal flip. The training accuracy drops to 93.7% and the test accuracy is lower than all Mixup-only tests summarized on Table 4. This indicates additional data augmentation may add excessive regularization. Among all the Mixup tests, we find $\alpha=0.2$ leads to the best model.

We perform an additional test of the final Mixup model with $\alpha=0.2$ on a real-world dataset containing 7,200 spray images. Unlike the training set where spray images are collected from multiple sources, this dataset only contains images from one test, albeit multiple testing conditions were covered. The classification results are manually examined by engine spray experts and performance measures are reported on Table 5. Note that this dataset does not contain any image in the Pre/Post class so the corresponding measures are not available.

About 64% of the tested spray images belong to the no collapse regime and they are all correctly detected by the classifier. This is crucial to the engine experts as no collapse sprays are desired for optimal engine performance and emissions, and hence correctly detecting this regime would allow them to use the corresponding testing conditions (ambient pressure, fuel temperature, injector geometry, etc.) for further optimization. On the other hand, only 12.7% of the test images belong to the collapse regime and the model is able to accurately predict 96% of them. The precision score of collapse is relatively low (97%) as well, indicating the model struggles with some images that possibly belong to the ”transitional/collapse” regime shown in Figure 1 (d). Nonetheless, the model reaches an overall accuracy of 98.7%, which is 0.7% higher than the test accuracy on Table 4.