Microstructure Surface Reconstruction from SEM Images:
An Alternative to Digital Image Correlation (DIC)

We study 3D micsrostructure surface modeling over time for a material undergoing fatigue testing. Fatigue refers to the initiation and propagation of cracks in a material due to cyclic loading. The experimental study of fatigue at the microstructure level is often done through visual inspection of test materials using scanning electron microscopes (SEMs).

The scanning electron microscope (SEM) is the most commonly used tool for imaging of microstructures . These are structures with dimensions around 1 micron ($10^{-6}$m). The image formed is a projected one, similar to traditional optical cameras.

Figure (1) from [1] shows an example of such a study. A small rectangular block of nickel (roughly 10 $\mu$m $\times$ 30 $\mu$m) is placed under the SEM and repeatedly subjected to cycles of tension and compression in the lengthwise direction ($y$ direction in the image). Over time cracks begin to form at internal planes at certain known orientations that relate to the crystal structure of the material.

The images show the growth of these cracks in both number and size. It is of interest to material scientists to understand the deformation of the surface which can be mapped to a strain tensor. Material Scientists then use the strain tensor to predict material life and failure modes. One technique commonly used today is Digital Image Correlation [3]. In this technique, shown in figure (2), a known so-called ’speckle’ pattern is imprinted on the sample being tested and images are correlated over time to estimate how the surface is deforming. This technique, though, is invasive and limited to 2D estimates of the surface deformation.

We present an alternative approach that estimates the magnitude and shape of the extrusions and intrusions on the surface – in other words the depths and heights of the peaks and valleys relative to the surface, in addition to the lateral (in-plane) deformation. The surface extrusions/intrusions can more accurately inform the correlation between fatigue life and slip localization. Recent studies suggest that there is a direct correlation between those and that fatigue life can be predicted from these out-of-plane displacements. Our goal is to detect variations in the z-direction on the order of 0.1 $\mu$m.

During the experiment, the SEM cannot be re-oriented so all the images obtained are from a fixed head-on viewpoint. Once the experiment is complete, though, the SEM can be re-oriented to obtain additional views.

Our approach is summarized as follows. First we use stereo metrology to estimate the surface roughness at the end of the experiment using two images of the sample at the end of the experiment from different camera orientations. Then we use optical flow techniques applied to single images of the sample during testing to estimate the lateral (in-plane) evolution of the cracks.

3D imaging of microstructures is possible directly through other microscopy techniques, such as scanning tunneling microscopes (STMs) and atomic force microscopes (AFMs). The former, though, requires extremely thin materials (on the order of nm’s), and the latter is limited to depths of $\sim 5\mu$m and is costly. Therefore there has evolved a relatively rich set of literature exploring the reconstruction of 3D models from 2D SEM images in recent years. These include the studies of [4, 5, 6] for example. These studies produce disparity maps from stereo metrology, but do not specifically study the geometry of cracks.

Also, as mentioned earlier, the technique of Digital Image Correlation [3], is commonly used to study 2D surface deformation of samples undergoing fatigue testing.

Our contribution (to our knowledge) is thus to offer an approach that combines stereo imaging from the end of the experiment, with non-invasive optical flow analysis applied to mono imaging during the experiment to provide a complete 3D view of the deformation of the sample undergoing fatigue.

All the images were provided by Prof. J. El-Awady, Associate Professor of Mechanical Engineering at John Hopkins University, co-author of [1] (and my brother) [7]. The images are all original acquisitions from his lab.

Now that we have an estimate of the surface roughness in the form of point depth estimates derived from stereo views of the final sample state, we consider the evolution of these points from the original undeformed state. This will provide insight into the lateral (x-y) movement of the points over time. Recall that while the experiment is being conducted we only have access to a single fixed front-view image of the sample.

We use the Lucas-Kanade Opticalflow method to estimate the lateral movement associated with the microcracks. We employ it using two separate sets of initial features to track. The first uses features identified using the Tomasi-Shi corner detection approach. While we will not have depth estimates for these points, they nonetheless provide a baseline for comparison of estimated lateral movement. Then we will use our corresponding points with depth estimates from section 4.2.

We show how to reconstruct a 3D model of surface roughness from SEM images of a material undergoing fatigue testing. During the experiment, the camera (SEM) is fixed and produces only frontal views of the sample. At the conclusion of the experiment the sample can be detached and multiple views (at higher resolution) can be obtained. We use the views at the end of the experiment to reconstruct a 3D model of the surface roughness. Then, we use optical flow techniques to propagate the final surface back in time to obtain an estimate of the lateral movement of the points as well. Together the lateral and depth movement provide a comprehensive understanding of the surface deformation of the sample.

Experts in the field have indicated that all our estimates of both out-of-plane deformation as well as in-plane lateral deformation are of a magnitude and shape consistent with theoretical expectation. Thus it represents a promising non-invasive alternative to the current state of the art, Digital Image Correlation, which requires imprinting a speckle pattern on the sample, and is limited to just 2D estimates unless it is augmented by further costly measurements such as x-ray tomography.

A number of next steps are warranted:

• •

  The estimates should be confirmed experimentally. Tools such as atomic force microscopes can be used to obtain a measurement of the surface roughness. Samples can also be sectioned and examined to obtain depth measurements. While this is typically not done due to the cost and often limited access to such tools, the promise of the technique presented warrants performing this work.

• •

  Once the images are obtained, the most tedious aspect was manual selection of corresponding points. An investigation of automating this process should be undertaken. Tomasi-Shi corner features appears a promising approach as a starting point for feature identification. Our experiments suggest it will select prominent features along the cracks much like I did visually. We should investigate how this can be used as part of an automated process (or at least semi-automated) to identify the corresponding points.

• •

  Finally the surface deformation should be mapped to a strain tensor, which is the physical quantity that material scientists are ultimately seeking.

The author would like to thank Andrey Kurenkov for his assistance and insightful suggestions about areas of investigation. Thank you, Andrey, for generously sharing your ideas and guidance.

Further, all the data was courtesy of Prof. J. El-Awady of John Hopkins University. Prof. El-Awady provided the motivation for the work and also generously assessed, reviewed, commented on, and validated the relevance and quality of the results.