Grain and Grain Boundary Segmentation using Machine Learning with Real and Generated Datasets

The coding for the manual thresholding was done using OpenCV opencv_library . All of the code for this work is available on github and kaggle PW . A step by step approach to manual thresholding can be seen in Fig 2. Step 1 is to collect the microstructure image. The material in this work is 316L stainless steel printed on an EXone printer from an optical microscope at 500X. Step 2: Convertion of image to black and white. Step 3 is a pixel intensity evaluation for pixel cutoff determination. The histogram shown for step 3 is for this purpose, and the X-axis is pixel intensity and the Y-axis shows the number of times that pixel intensity has occurred. The red line on this histogram shows the selected thresholded value, also called the pixel cutoff value. Step 4a and step 4b are both examples of thresholding. Step 4a is an example of standard thresholding where all of the pixels ($>$130) are set to 255 (white) and all pixels ($\leq$130) are set to 0 (black). The value 130 is shown in red on the histogram. Step 4b is an example of an adaptive gaussian threshold. The threshold value is a gaussian-weighted sum of the neighbourhood values minus the constant C. The neighborhood values are a block of pixels, for this case it was (55x55), and the constant subtracted was 2.

In step 5 a median blur is applied 2 times to a 5x5 pixel area, and then 2 more times to a 7x7 pixel area. In step 6 image segmenation is applied, and the grains are seperated from the grain boundaries. In step 7 the grains are filtered, usable grains are shown in green and unusable grains are shown in red. Unusable grains are grains that are too small or grains that have too much perimeter per area. Too much perimeter per area indicates the grain did no get properly segmented during the total process. Essentially two or more grains are connected by a small ”neck” region.

Image segmentation via threshold is a simple but highly customizable process. There are several steps in this process where the user has to select a value and check the results. For example, threshold cutoff, block size for blurring, block size for the adaptive gaussian method, etc. All of these values lead to different results, so the user must pick the one that looks the best in order to automate the grain segmentation process. Different materials, imaging equipment, or image settings can effect the images, and could easily require different variables for optimal segmentation.

The gradient based approach identifies sharp changes in pixel intensity, and can classify those areas as an edge. The first and second derivatives of pixel intensity as it relates to pixel position are used to determine the gradients. The method used in this work is the ”Canny Edge Detection Method,” developed by John Canny canny1986computational . First the image is converted to greyscale and then gaussian filters are used on the pixel values to slightly blur the image. This will reduce the amount of predicted edges in the result, which is helpful because the gradient based approach tends predict too many edges. After the gaussian filters, the pixel intensity gradient is calculated (first and second derivatives). Lower changes in pixel intensity are filtered out, unless they directly connected to a high change in pixel intensity.

The application of the Canny Edge Detector can be seen in Fig 3. First the image was converted to black and white. Next the Canny Edge Detector was used. In the third step, two Gaussian blur iterations were applied to a 3x3 pixel block. In the fourth step, the image is thresholded into 0 and 255 (black and white). The threshold value was very high (200), meaning most gray pixels were set to black. Step 5: segmentation occurs. Step 6: Filtering of the data occurs. The same filters were used as the previous section. Too much perimeter per area were filter out, and too little a grain size were also filter out. Image segmentation via gradient detection is a highly customizable process and optimal parameters can be difficult to determine. Small adjustments to pre and post filters will cause significant changes in the final predictions. The engineer or user of this method must tailor the settings and parameters of the algorthm for specific materials or image settings.

The next detection method is deep learning-based method for edge detection in images. The architecture of the HED utilizes a fully convolutional network, which was trained end-to-end on a large dataset of images with ground truth edge maps.xie2015holistically CNN’s are particularly well-suited for tasks involving image processing and analysis, such as object detection, image classification, and image segmentation o2015introduction ; albawi2017understanding ; warren2021rapid .

In Fig 4, a clear step-by-step guide for utilizing the HED approach is provided. First the image is converted to black and white. Next it is fed into the HED model to determine the edges. The third step is to apply an adaptive guassian threshold to convert all pixels to either 0 or 255 (white or black). The final two steps are segmentation and data filtration. The HED model outputs a pixel intensity that represents the confidence that the pixel is an edge or not an edge. That is why step 3 (adaptive thresholding) is required. The HED is highly effective at detecting grain boundaries, but its suitability as an end-to-end solution is limited due to its tendency to also detect undesired features such as pores and polishing scratches. This is primarily because HED lacks the ability to distinguish between different types of edges, relying on low-level image features that generalize its edge detection capabilities to various image characteristics.

The most arduous way to segment the grains of image is the manual segmentation. This is done by the user manually annotating the grains or grain boundaries. For this work it was completed by marking the grain boundaries in a specific color, and then thresholding the image so that all annotated pixels are 0 (black) and all others are 255 (white). After that the same process of segmenting the image computationally and filtering out small particles occurs. This process is shown in Fig 5. All inaccuracies that occur in this method are from human error. Polishing scratches and pores must be avoided during annotation of the grain boundaries. This method cannot be automated, each image requires human attention. For this work all 480 images of grains were painstakingly manually segmented, and this was completed to allow the selected machine learning methods to train and replicate the process. All of the previously mentioned segmentation methods/algorithms were used (HED, Gradient, Threshold) on the dataset to provide benchmarks for each of the methods and the proposed methods in this work. All of this data is available on Github and Kaggle.

The grain boundary image dataset used in this study consists of 480 images, each with dimensions of 400 x 300 pixels. The grain intercept method, initially developed by Heyn and widely employed for rapid grain size measurements heyn1925physical , involves drawing a line across the image and tallying the number of grains intersected by the line. The average grain size is then determined by dividing the total number of grains by the line length, as shown in Equation 1. To automate this process and conduct measurements at each pixel, code was implemented for all the segmentation techniques outlined in Section 2, resulting in a total of 144,000 X-direction measurements and 192,000 Y-direction measurements. Table 1 presents the average grain size and standard deviation for both the horizontal and vertical directions. Although the grain intercept method provides a substantial number of measurements, it is inherently limited as a 1-dimensional approximation of a complex 3-dimensional structure. Nonetheless, its continued utilization in current research reinforces its relevance in this study.

The planimetric method, a 2-dimensional approach, enables the extraction of size and shape data regarding the grains. Extensive data from the segmented dataset was collected, which has been categorized into size data (Table 2) and shape data (Table 3). These datasets encompass all the segmentation methods listed in Section 2. The planimetric size data comprises four measurements. The first on is the total grains metric which represents the overall count of grains across all images. The second measurement is the average individual grain area, accompanied by its standard deviation. The third and fourth entries in Table 2 correspond to the total grain area per image and the total grain boundary area per image. These two values can be computed interchangeably using Equation 2.

The planimetric shape data encompasses five distinct measurements. Circularity represents a geometric property of a shape, ranging between 1 and 0, where 1 denotes a perfect circle and 0 signifies the theoritical opposite of a perfect circle. The closer the value is to 0, the rougher the 2-dimensional shape becomes. The equation for circularity can be seen in Equation 3. Another measurement is the average maximum diameter, which represents the maximum distance from one side of the grain to the other, passing through the centroid of the grain. The subsequent three measurements pertain to the X and Y directions. The first is the average width of the grains, denoted as the average X-diameter. The next measurement is the average diameter in the Y-direction, also known as the average height. The final value is the aspect ratio, obtained by dividing the average X-diameter by the average Y-diameter, providing a measure of the width-to-height relationship.

For assessing the segmentation model accuracy, the dice score and the IoU (Intersection over Union) score will be employed. When classifying pixels in a segmented image, there are four possible outcomes: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). The dice equation, represented by these variables, can be seen in Equation 4. Additionally, the probabilistic version of the equation is provided in Equation 5. These equations serve as quantitative measures to evaluate the performance of the segmentation models.

To facilitate implementation, it is crucial to establish and indicate a ground truth. In this study, the manually segmented images serve as the ground truth, with the only potential error arising from human inaccuracies during the grain boundary sketching process. Table 4 showcases the comparison of Dice and IoU scores for the manually thresholded method, the gradient method, and the HED method. These scores can serve as benchmarks, alongside other grain properties, to quantify the accuracy of the machine learning methods developed in this research.

A Voronoi tessellation pattern is generated by dividing a plane into regions based on the distances to specific points in the plane voronoi1908nouvelles . Each point, also referred to as a seed, corresponds to a region comprising all the areas in the plane that are closer to that point than to any other. This partitioning of the plane into regions defined by proximity to the seeds forms a Voronoi diagram. An example of a Voronoi diagram, utilized in this work, is depicted in Figure 6. The size of the Voronoi cells is determined by the area measurements obtained in the previous section. The average area of the Voronoi cells is approximately equal to the reciprocal of the point density. For instance, if the point density is 10 points per unit area, the average cell area would be approximately 0.1 units. Based on this calculation, the average grain (cell) area for the generated data was set to 1000 $\mu m^{2}$. Comparatively, the average individual grain area for the manually segmented images was found to be 1068 $\mu m^{2}$, so 1000 $\mu m^{2}$ was selected to closely represent the actual data.

To accurately replicate the challenges observed in the images, noise was intentionally introduced into the Voronoi tessellation pattern. The main challenges encountered in the images include polishing scratches and pores, as depicted in Figure 1. To incorporate scratches, random lines were plotted within the Voronoi graph, while pores were simulated by randomly placing dots of varying sizes. An illustrative example showcasing these defects can be seen in Figure 6. While these defects could be mitigated through more meticulous polishing techniques and improved etching methods, achieving perfection in these complex processes can be challenging, especially for certain metals. Therefore, employing machine learning methods to automatically disregard these defects represents a viable approach to mitigating their undesired effects during the measurement process.

The overall plot area ranges from 0 to 800, but the actual image spans from 200 to 600. Consequently, a 400x400 pixel image, identical to the experimental data, is created. This arrangement allows for the possibility of scratches either traversing the entire sample image or being confined within the image boundaries. During the generation of the artificial data, the quantity of pores and scratches was deliberately varied, and this will be further explained in the subsequent section.

The quantity of scratches within the entire plot area was determined using a uniform random number generator ranging from 2 to 10. The starting and stopping locations of the scratches were also randomly assigned using uniform random distributions for both the X and Y coordinates. Moreover, the number of randomly sized pores was generated using a uniform random number between 2000 and 4000. Additionally, random black pixels were inserted into the images, with the quantity determined by a uniform random number between 10,000 and 20,000. To impart a gray color to the grains, a subtraction operation was applied to all white pixels (255) using a normally distributed Gaussian distribution centered around $100\pm 10$. Detailed numerical values for all of noise distributions can be found in Table 5. The objective is to train a CNN to accurately predict the mask, representing the initial noiseless Voronoi tessellation, given a noisy Voronoi tessellation. Although a CNN can readily accomplish this task, additional noise will be introduced to enhance the robustness of the model.

To enhance the training process and develop a more robust CNN for image segmentation, the next round of noise was introduced by exploiting the limitations of two segmentation techniques mentioned in Section 2: Manual Thresholding and Gradient-Based Method. This approach aims to leverage the deficiencies of these techniques to further train the model. Specifically, the median blur technique described in Section 2.1 and the Gaussian blur technique discussed in Section 2.2 were implemented. Additionally, the inverse of an adaptive threshold, which is similar to an erosion technique, was applied. These techniques were utilized in various iterations and combinations to create a diverse range of noise patterns. The specific application method can be found in Table 6. For a visual representation of the generated noise applied to an original Voronoi tessellation for each set in Table 6, refer to Figure 7.

The U-Net architecture, introduced in 2015 by Ronneberger et al. ronneberger2015u , has emerged as a powerful framework for pixel segmentation tasks. Its design is particularly well-suited for this purpose as it effectively captures both local and global features present in the input image. One key advantage of U-Net is the incorporation of skip connections, enabling the model to leverage both high-level and low-level features during the segmentation process. In many image classification problems, both levels of features contain crucial information necessary for accurate classification.

In the current era of Artificial Intelligence growth, numerous variations of the classic U-Net model have been proposed. However, to maintain consistency and facilitate benchmarking, we have opted to employ the standard U-Net architecture in this study. Comparing the performance of different models can be easily accomplished since all the relevant data is publicly available.

For the activation functions within the layers, the Rectified Linear Unit (ReLU) function was utilized. The final activation function for the classification task was the sigmoid function. The optimizer function employed was ADAM, while the loss function used was binary cross-entropy. During training, a batch size of 16 was utilized, and the training process consisted of 50 epochs. A validation split of 10$\%$ was employed to monitor the model’s performance during training. The layout of the U-Net architecture employed in this study is depicted in Figure 8.

The dataset was split into 80$\%$ for training and 20$\%$ for testing. Two crucial aspects of the training process are determining the training data and defining the desired outcome, referred to as the mask. The mask represents the specific pixels classified, in this case classified as gran boundaries. In this study, the mask (grain boundary classified pixels) are assigned a value of 0 (black), while all grains are assigned a value of 255 (white). The masks used in this work were primarily manually segmented by human experts, except for the Voronoi-generated data, where the original unaltered Voronoi diagram serves as the mask. An example of such a mask is illustrated in Figure 6.

The total composition of the training data is a critical aspect to consider. The obvious approach is to train the model solely on manually segmented data. However, the question arises: would incorporating the Voronoi tessellation-generated data enhance the model’s robustness? This question will be addressed in the results section. The different compositions of the training sets are depicted in Figure 9. The first five compositions involve varying percentages of generated and manually segmented data, allowing for a comparison between the Voronoi method, the Manual method, and a combination of both.

Training sets 6-10 follow a more intricate approach. Traditional segmentation methods from Section 2.1, 2.2, and 2.3 were utilized to preprocess the data. This means that the mask remains the same (manually segmented image), but the original grain image undergoes modifications through the application of the manual threshold method, the HED method, or the gradient method. Subsequently, the U-Net model is trained to map from the manually segmented image to the mask (manually segmented image). This approach was employed to explore whether it would lead to a more robust model and what effect the incorporation of this data would have on the accuracy of the model.

The Dice score is a widely used metric in the machine learning and computer vision communities to assess the overlap between two sets of data, making it particularly suitable for evaluating image segmentation methods. In this work, the Dice score was employed to compare the performance of different methods on the various training set compositions depicted in Figure 9. The Dice score ranges from 0 to 1, where a score of 1 indicates a perfect overlap between the predicted segmentation and the ground truth. In this study, the focus was on accurately classifying pixels as grain boundaries, and thus the Dice score was calculated based on the ability of the models to accurately identify grain boundaries.

The Dice score results are presented in two graphs in this section. Figure 10 displays the Dice scores for all the traditional methods described in Section 2. Figure 11 depicts the Dice scores for the different U-NET training compositions utilized in this work. It is important to note that the manually segmented image is considered the ground truth in this study, which is why it achieves a perfect score in both figures. The Dice score graphs provide valuable insights into the performance of the segmentation methods and their ability to accurately identify grain boundaries.

In the graph in Figure 10, the traditional segmentation methods displayed relatively low Dice scores, with the HED method performing unexpectedly poorly. However, these methods can still be valuable for generating noise to enhance the robustness of machine learning models, as demonstrated in this study. The lower scores emphasize the challenges and limitations of manual and rule-based approaches in accurately capturing complex grain boundary features. The integration of machine learning techniques, like U-NET, offers improved segmentation by capturing both local and global features more effectively.

Figure 11 illustrates that the dice scores for the U-NET models are consistently around 90$\%$. Surprisingly, the inclusion of fully artificially generated data in training sets 2, 3, and 4 did not adversely affect the scores. Even when using solely artificially generated data in training set 5, the dice score only dropped to 87$\%$, which is still remarkable considering the absence of real grain images in the neural network’s training.

Furthermore, the dice scores for training sets 6 through 10 demonstrate that incorporating noise generated from the traditional methods does not negatively impact the scores. This suggests that such noise can be utilized to train more robust models that are resistant to the effects of traditional segmentation methods. Notably, training set 10, which included noise from all three traditional segmentation methods, achieved the highest dice score.

The accuracy of grain geometry measurements is of greater interest to the materials science community than the accuracy of pixel classification. To evaluate the accuracy of the methods used in this paper, the standard error equation (Equation 6) is employed. The error values are depicted in Figure 12, where each axis represents a specific grain measurement, such as circularity, average maximum diameter, average X-diameter, average Y-diameter, grain intercept method in the X-direction, grain intercept method in the Y-direction, average grain area, and total grains.

The error values are calculated by comparing the average values obtained from the grain measurements of the different methods to the corresponding average values from the manually segmented images, which are regarded as the ground truth. The colorbar in Figure 12 indicates whether the estimated values overestimate (blue) or underestimate (red) the true values. This analysis provides insights into the accuracy of the grain geometry measurements, highlighting the extent to which the methods over or under predict the values.

Based on the values in Figure 12, the traditional methods (manual threshold, Gradient Approach, and HED Method) exhibit higher error percentages for all grain measurements compared to the other methods. Among the traditional methods, the manual threshold method shows relatively lower errors. However, both the Gradient Approach and HED Method significantly overestimate the total number of grains in the images, leading to skewed values in other measurements. The average maximum diameter, average X-diameter, and average Y-diameter are also affected by overestimation, although the manual threshold method has lower errors in these measurements compared to the other traditional methods.

In contrast, the UNET training set variations generally exhibit lower errors. Training set 5, trained entirely on artificial data, shows the highest error with an underprediction of grain count by approximately 38$\%$. This is attributed to the presence of incomplete boundaries and small neck regions connecting grains, as evidenced by the lower circularity values. Training set 1 performs well across all measurements, and the inclusion of artificial grains in sets 2, 3, and 4 slightly increases the error percentages. However, these sets outperform the traditional methods and some other training sets. Training set 10, which includes noise from the traditional methods, also demonstrates good performance, indicating the beneficial impact of incorporating diverse noise sources in the training process. Overall, the UNET-based methods outperform the traditional methods, with different training set compositions yielding varying levels of accuracy in grain measurements.

The qualitative results provide a closer examination of the approach by presenting individual image results in grain boundary segmentation. This allows for a detailed analysis of the segmentation performance on a case-by-case basis.

Figure 13 presents example images from the UNET trained on data set 3, which comprised 50$\%$ real grains with manual segmentations and 50$\%$ artificially generated grains using the Voronoi tessellation method. Image 3 shows the UNET’s prediction of the original mask (image 2) based on the input image (image 1) from the testing set. The UNET successfully predicted the mask despite not seeing this specific image during training.

Furthermore, the same UNET trained on data set 3 demonstrated its ability to accurately predict grain boundaries in a real grain image. Image 4 represents a real grain image, image 5 shows the corresponding manually drawn segmentations, and image 6 displays the UNET’s prediction of the grain boundaries. The high degree of similarity between images 5 and 6 highlights the exceptional performance of neural networks in grain boundary segmentation tasks. These results showcase the superior capabilities of the UNET model trained on data set 3 in accurately predicting grain boundaries, even when presented with unseen images during training.

In Figure 14, the results of a UNET trained exclusively on artificially generated data (training set 5) are presented. This UNET model was never exposed to actual grain boundary images but was solely trained on data generated using the Voronoi tessellation grain image generation method proposed in this work.

Image 1 in Figure 14 displays a real grain image of stainless steel, with the predicted grain boundary mask overlaid in green. Despite not being trained on this specific data, the UNET was able to provide a reasonably accurate estimation of the grain boundaries. For comparison, image 4 shows the manually drawn grain boundaries for the same grain image. Images 2 and 5 show the masks without the grain boundaries, provided for reference. Although the predictions from UNET trained on data set 5 are not perfect, they demonstrate that the model successfully avoided classifying voids and polishing scratches as grain boundaries.

Images 3 and 6 showcase stainless steel images produced through injection molding, which were not used for training and were not captured specifically for this work. These images serve to demonstrate the potential of using artificial data for training more robust models. The UNET trained on data set 5 achieved reasonably accurate grain boundary predictions for these images as well, even in the cases where some grains appeared darker, like in image 6.

The summary of results from this work will be broken down into the different approaches to keep the statements organized and keep the clarity of each statement intact.

The Traditional Methods: The traditional methods (manual thresholding, gradient-based approach, HED method) require parameter tuning and are not universally applicable. However, they can be used to generate noise for training more robust models.

UNET Training Set 1: Training set 1 performed well on both the dice score and the actual grain measurements, as should be expected.

UNET Training Sets 2-4: Training Sets 2-4, which included artificially generated grains, maintained high scores in dice score and grain measurement accuracy. Artificial data can effectively increase training data without compromising accuracy.

UNET Training Set 5: This set is composed entirely of artificially generated data, and performed reasonably well, surpassing the traditional methods. The voronoi grain generation method can be used to supplement training data for grain segmentation.

UNET Training Set 6-10: Training Sets 6-10, incorporating noise from traditional methods, achieved high scores in dice score and grain measurement accuracy. Training Set 10, which included noise from all three traditional methods, performed the best. Incorporating noise from these methods enhances model robustness.

Planimetric and Line Intercept Method: Planimetric analysis yielded more accurate grain measurements compared to the line intercept method. Deep learning-based image segmentation can provide suitable images for planimetric analysis.

In summary, the deep learning approach using UNET models demonstrated superior performance compared to traditional methods for grain boundary segmentation. Incorporating artificially generated data and noise from traditional methods can enhance model performance and robustness. Additionally, planimetric analysis provides more accurate grain measurements than the line intercept method.