A Machine Learning Approach to Robustly Determine Director Fields and Analyze Defects in Active Nematics

The nematic tensor, or the $Q$-tensor (Bantysh et al., 2023; Golden et al., 2023; Doostmohammadi et al., 2018; Keber et al., 2014; Ramaswamy, 2010; Doostmohammadi et al., 2016), describes the director field; that is, the degree of alignment and the direction of the local nematic orientation. A reliable director field is essential to accurately detect, measure, and track defects in active nematic systems. One way to calculate the director field from experiments is to use the PolScope (Oldenbourg et al., 2005; Zhou et al., 2014; Shribak & Oldenbourg, 2003), which is a powerful tool for visualizing and analyzing the dynamics of nematic structures. It operates by shining polarized light through the specimen, which interacts with the aligned molecules or particles in the nematic phase. This interaction alters the polarization of the light, allowing the PolScope to measure birefringence quantitatively at every single pixel in the image. The resulting retardation of light allows the PolScope to accurately determine the degree and orientation of microtubule alignment, pixel by pixel, providing important insights into the structural characteristics and behavior of liquid crystal systems. PolScope generates a pair of images - the retardance image and the corresponding director field image, at any given time point. The calculated director field enables a direct comparison between experimental observations and theoretical results. This data also can be used to train dynamics prediction models, such as proposed by Zhou et al. (2021); Colen et al. (2021). Nevertheless, certain experimental setups and controls are not feasible using PolScope. For example, some experiments use light to control the dynamics of active nematics (Zarei et al., 2023; Zhang et al., 2021; Lemma et al., 2022; Ross et al., 2019). These experiments illuminate the sample from above, which makes the traditional PolScope setup impractical. In addition, high numerical aperture condenser and objective lenses typically introduce distortions that limit the purity of polarization in a light microscope. These distortions present challenges and limitations when using a PolScope (Oldenbourg et al., 2005). Lastly, the nematic material’s small retardance values are often overshadowed by stronger background birefringence from microscope objective strain and Fresnel reflectivity. Although background correction is a standard practice, it doesn’t fully account for the microscope slide’s birefringence and is sensitive to variations across the image, optical alignment, and temperature. This can result in inaccuracies in local orientation measurements in some regions. For example, discontinuous director fields were observed in the middle of microtubule bundles, see Figure 2. This may result in false detection and inaccurate localization of defects.

Alternatively, other instruments can offer more flexible control and design of the experiments, but cannot generate the corresponding director fields directly. For example, a fluorescence microscope enables direct light control (Zhang et al., 2021; Lemma et al., 2022; Ross et al., 2019) or ATP control of the system’s activities, but does not offer the capability to extract director fields. In some experiments, specialized fluorescent dyes or proteins are added at low concentrations to enable the visualization and tracking of individual microtubule movements. However, such methods only provide sparse director field information that is randomly distributed across the whole view. In these situations, one can use image processing methods to extract orientations from raw images directly from experiments, such as fluorescence images. Usually, an in-house method is developed using softwares such as MATLAB for extracting director fields from raw images (Tan et al., 2019; Shi & Ma, 2013; Blackwell et al., 2016; Duclos et al., 2020; Rezakhaniha et al., 2012). The methods follow a similar approach by first finding the gradient change of a raw image, then coarse-graining the result, and finding the smallest eigenvalue. More details are presented in Appendix A.1 and Ellis et al. (2018). In this work, we use one of the existing implementations of traditional methods developed by Michael M. Norton³³3Relevant code can be found on GitHub: https://github.com/wearefor/qcon_nematicdefectfinder, referred to as the “TM” in the rest of this paper.

However, the effectiveness of the TM can be limited because raw images contain complex signals arising from variations in experimental settings, noise, limitations of microscopy, etc. Examples of such limitations include the following. First, the analysis of retardance images is intrinsically difficult at regions that have low contrast, are out-of-focus, or have distortions due to a variety of effects. For example, although active nematics are assumed to be two-dimensional by imaging systems, they actually move in a shallow three-dimensional container. This three-dimensionality can influence both the illumination and the focus of the microscope, thereby altering the pixel intensity in the resultant images. Additionally, active nematic filaments, when compressed by adjacent ones, can buckle into a U shape, causing a fissure that exposes the base of the container. Such fissures often run counter to the direction of movement, which can result in a miscalculation of the local orientation. We show some examples of orientation miscalculation by the TM in noisy regions in Figure 3. Finally, the inherent variability in active nematics systems and the varying levels of activity observed across different experiments contribute to a diverse distribution of noisy regions across successive frames.

Second, the TM may require extensive manual adjustments of its parameter settings to make it work satisfactorily for different images. In other words, raw images from different experiments or conditions cannot use the same parameter setting. Sometimes, manual adjustment is needed for different images in the same video (see Figure 4 for an example). This greatly hinders large-scale reliable studies of active nematics. Many factors contribute to this challenge, such as: 1) active nematic materials can exhibit significant variations even under identical experimental conditions; 2) different imaging instruments or settings can lead to different noise distributions; that is, different characteristics or statistical patterns of errors in the acquired images; 3) the behavior of nematic materials may not be consistent across the sample (e.g. due to spatially varying activity levels or boundary conditions, which may be according to design or arise from experimentally uncontrollable variations), resulting in defects with varying sizes and mean separation distances that require different configurations of window sizes.

To tackle the above challenges, we developed a machine learning based technique to robustly and automatically extract director fields from raw images. Machine learning has already made significant impacts on active nematics (Cichos et al., 2020; Rabault et al., 2017; Hannel et al., 2018; Muñoz-Gil et al., 2020), and more applications are being explored. For example, we developed a machine learning technique for learning and predicting dynamics of active nematics (Zhou et al., 2021), which however relies on the availability of director fields. Using our proposed machine learning approach, not only can we reliably and robustly calculate the director fields from raw experimental data directly, but we also enable many downstream tasks such as defect detection, defect tracking, and dynamic prediction. These tasks are significantly simplified, compared to previous approaches, for raw data that does not have the director fields readily available. We benchmark our machine learning model against our in-house implementation of an image processing method (TM) that follows a similar approach to other existing methods(Tan et al., 2019; Shi & Ma, 2013; Blackwell et al., 2016; Duclos et al., 2020; Rezakhaniha et al., 2012; Ellis et al., 2018). This method takes a gray-scaled image as the input. After denoising the raw image by Gaussian Blur with standard deviation $\sigma$, it assumes that image intensities change most dramatically in the direction perpendicular to the flow of the microtubule bundles at each point. Thus, for each pixel, it calculates the intensity change matrix using convolution, which is then decomposed to obtain its eigenvalues. The local orientation centered at a specific pixel is indicated by the eigenvector associated with the smallest eigenvalue. Once the $Q$-tensor is obtained, topological defects can be identified by calculating the winding number (Mermin, 1979) of each point. This is typically done by applying convolutional filters (e.g., Gaussian Kernels) of user-defined window size on each pixel on the director field obtained. More details of this method can be found in Appendix A.1. Compared with the TM, the director field information obtained by our method is more accurate, which leads to better defect detection results. Figure 5 shows an example of defects that are missed by the TM. These defects can be recovered using our machine learning models. We note that the TM defect detection results differ slightly when different parameter settings are chosen. Fig. 20 (Appendix A.6) shows examples of changing the parameter settings for individual frames. However, the TM results are consistently less accurate than those obtained by our machine learning approach, and we have tuned the TM parameters for each dataset to achieve reliable results for all frames in that dataset. In addition to increased accuracy, a key advantage of the machine learning approach is that it requires less parameter tuning.

In summary, the main contributions of our work include the following:

• •

  We developed a robust machine learning approach to accurately extract director fields from raw microscopy images of active nematics. Our approach is able to handle raw images captured under various experimental settings (e.g., lighting, activity levels of active nematics, microscope lens, etc.).

• •

  We used a masking strategy to correct some regions of director fields calculated by the PolScope that are inaccurate due to sub-optimal retardance images, such as over-/under-exposure, out-of-focus regions, etc.

• •

  We demonstrate promising downstream applications using the director fields generated by our model, including defect detection, defect orientation, defect tracking, etc.

• •

  To facilitate defect tracking as a downstream task, we utilized a matching algorithm across several consecutive frames and evaluated outcomes across frames. This approach enabled us to enhance tracking accuracy by filling in gaps for absent defects in a particular frame, while not in others.

In most cases, PolScope works well in capturing the retardance images of active nematics and correctly calculating the corresponding director fields. However, PolScope occasionally produces sub-optimal local orientation calculations (examples in Figure 2), which can hamper the efforts to build a machine learning model to extract director fields. Fortunately, such errors can be automatically corrected using machine learning. Specifically, we adopted the Masked Image Modeling (MIM) (Xie et al., 2022) method (see Figure 6 Top) that randomly masks out patches in an image decided by a mask size and a masking ratio, and trained a machine learning model to reproduce the masked-out patches using the rest of the image. We chose the mask size as 10x10 pixels and the masking ratio as 30% of the whole image. To make this selection, we tried several different mask sizes (6x6, 10x10 and 20x20 pixels) and different masking ratios (20%, 30% and 50%), and found the chosen setting produced the lowest Mean Absolute Error (MAE) of 0.0241. Once trained, the MIM model can be applied to improve director fields not used in training. Two examples are shown in Figure 6 Bottom to visualize the effects of this process.

Using the corrected director field as the target output, we can train our machine learning model to learn the proper mapping between the retardance image to its corresponding director field. Figure 7 illustrates the architecture of our machine learning model, which consists of a Gabor filter (Fogel & Sagi, 1989; Bianconi & Fernández, 2007) layer followed by a ResNet structure (He et al., 2016). Gabor filters are a set of filters widely used for texture analysis. Each filter can be tuned to detect spatially local signals with certain frequency components in a certain direction at a certain scale. One can use an array of Gabor filters to help perform multi-scale analysis of spatial patterns in active nematics, such as defects, vortices, or other patterns. More technical details about Gabor filters are provided in Appendix A.2. In our model, we used two sets of Gabor Filters, the first set has a smaller wavelength that aims to detect higher spatial frequencies and finer details, while the second set has a larger wavelength, aiming to capture global features. For each set of filters, 64 filters are used, each with a different combination of 8 orientation values and 8 scale values. For the ResNet, 16 blocks are used, as shown in Figure 7. Among the 16 ResNet blocks, 4 different channel dimensions are used: 64, 128, 256, and 512. More detailed configurations can be found in Appendix A.3.

One can detect defects by computing the winding numbers of individual points in the director fields (Mermin, 1979) by counting the local orientation changes centralized at each point within a predefined window size. Nevertheless, the director fields are discretized at the pixel level, which can lead to inaccurate winding number calculations. Some defects may not be detected even from more accurate director fields such as those calculated by PolScope (see Figure 18 in Appendix A.6) and our machine learning approach (see Figure 8, Left), due to various factors. These include over-exposed regions in the raw image, leading to an over-smoothed director field; or regions with high local defect densities, where the distance between the defects is smaller than the window size when calculating the winding number. The director fields calculated by our method significantly improve defect detection results (see Figure 11 in the Experiments section). However, in a non-equilibrium system of active nematics where speed and density change, it is possible that a defect is detected in one frame but missed in the next (see Figure 8 Right). This can cause the defect tracking algorithm to lose track of detects in consecutive frames and result in inaccurate measurement of defect annihilation and proliferation rates.

To tackle this problem, we match defects detected in three consecutive frames (at time points $t-1$, $t$, and $t+1$) using a graduated assignment algorithm (Gold & Rangarajan, 1996), and use the matching results to recover missing defects in the $t$-th frame. The coordinates of the rescued defects are interpolated using the coordinates of their matched defects at time $t-1$ and $t+1$. This allows for more accurate defect detection and tracking over time, see Figure 12, Figure 13, and Figure 14. Please refer to Section 3.3 and 3.4 for more results and discussions.

Finally, we trained a model with four ResNet blocks (32, 64, 128, and 256 channels, respectively) to estimate the orientation of a defect. The input to the model is a 64 $\times$ 64 patch centered at a defect, and the output is the angle of the defect. The model error is also measured by the Mean Absolute Error (MAE) between the estimated angles and the manually labeled ground-truth.

Figure 9 illustrates the advantage of our machine learning model over the TM on the PolScope retardance images. Using the polished director fields calculated by PolScope as the ground-truth, our trained machine learning model extracts more accurate director fields compared to the TM. We tested our model on 120 randomly selected retardance images excluded from the training set. Using the corresponding director fields calculated by PolScope as the ground-truth, we calculated the error of our model and the TM method. Our approach achieves a mean absolute error (MAE) of 8.8 degrees per location with a standard deviation of 1.46. As a comparison, the MAE of the TM approach is 18.1 per location with a standard deviation of 2.6.

The proposed machine learning approach is trained using PolScope retardance images and calculated director fields. However, once trained, it can be applied to raw images produced at other magnifications, by other microscope instruments, or under different experimental settings (e.g., activity levels or boundary conditions). There are various methods to control the behavior of active nematics, such as changing the concentration of ATP and motor proteins, applying hard boundaries, etc (Keber et al., 2014; Opathalage et al., 2019; Thijssen et al., 2021; Zhao et al., 2020; Hardoüin et al., 2019; Lemma et al., 2019). However, these methods control activity with only limited precision, and do not enable spatially varying activity profiles. To enable greater control over activity and the ability to change activity in space and time, researchers have developed light-sensitive motor proteins (Zhang et al., 2021; Lemma et al., 2022; Ross et al., 2019), with which the active nematic’s activity and dynamics can be regulated by varying the spatiotemporal patterns of applied light intensity.

To create an active nematic, we combined microtubules, kinesin motors, ATP, and bundling agents, and sediment the mixture on an oil-water interface. In the light-activated active nematic, the kinesin motors are equipped with light-sensitive domains, namely “iLID”and “micro”. When exposed to blue light, these domains bind together rigidly, creating a light-sensitive cluster that generates relative sliding for anti-parallel microtubules. To illuminate the sample, we used a Digital Light Processor (DLP) projector that projected 460 nm wavelength light from above onto the sample and we took data using a fluorescent microscope. However, the PolScope setup, which involves a universal compensator and an analyzer for circularly polarized light, limits access to the stage and does not allow us to use the DLP mounted to the stage for light-activated experiments. Additionally, data acquisition with PolScope requires taking 5 images in different polarization settings to calculate the retardance image and its director field (Shribak & Oldenbourg, 2003), which limits the frame rate for taking data. For example, with a camera exposure time of 500 ms, it takes 670 ms to capture a single image and roughly 3.4 sec for five images. The impact of these exposure times on the observations becomes noticeable when the nematic speed is high, as the motion between frames is difficult to document, and consequently, tracking defects becomes less precise. While the PolScope microscope has some advantages, such as calculating the director field without the need to label the proteins, it is not suitable for all experiments, such as the light-activated active nematic. Hence, it is desirable that our director field extraction model, which was trained on PolScope retardance images, can generalize well to other raw images. Our proposed machine learning model performs well on fluorescence images of active nematics with different microscope magnifications (see Figure 10) in an unconfined 2-dimensional experimental setting. Moreover, our model also generalize well to confined containers with different boundary conditions (see Figure 17 in Appendix A.6, and videos in the supplementary material). The authors highlight that even in the presence of unbound microtubules floating along the z-axis of the 3D container, which is not aligned with the local orientation within the 2D nematic system under investigation, the machine learning algorithm adeptly filters out these elements, ensuring reliable calculation of the director field.

The defect detection results using the director fields extracted by our approach are substantially better than those obtained using the director fields extracted by the TM (details in Appendix A.1). Figure 11 visualizes the defect detection results from raw images in one experiment using fluorescence microscope, which was not used to train our director field extraction model.

To evaluate the defect detection results, we manually labeled defects in 40 fluorescence images (not used in model training) from one experiment as the ground-truth. Fluorescence microscopy enables a light-responsive system that allows researchers to combine a considerable quantity of ATP with microtubules and regulate its behavior using light intensity. When the light intensity is increased, a larger number of motor proteins and microtubules become attached, leading to increased system activity and ultimately resulting in more defects. Figure 12 and 13 show the performance of our approach in both low and high light intensity experiments. Note that the high-intensity sample (Figure 13) has not reached steady-state, and thus the number of defects is increasing over time. Using the director fields extracted by our machine learning model, the numbers of defects closely track the hand-labeled ground-truth. Moreover, we closely observed the defect location generated by both methods. Compared with the manually annotated location, if the detected location is within a 40-pixel radius, which is roughly half of the averaged defect size (100um, 77 pixels), it is considered a precise detection. The size of the raw image is $1296\times 1280$. Figure 14 compares the percentage of precise detections using both methods. The director field extracted by our machine learning model enables better defect detection results (both defect detection rate and location).

Once defects are detected, we can track them over time. This can be achieved by matching the locations of defects across different frames using the graduated assignment algorithm (Gold & Rangarajan, 1996). This article proposed a fast and accurate graph matching algorithm using nonlinear optimization and graduated non-convexity to avoid poor local minima. This algorithm allows matching between graphs of different node numbers. This fits well with our system because it enables handling the spontaneous birth and annihilation of defects. However, it is possible that one defect is detected in one frame, but disappears in the next frame. This could be due to defect annihilation or a mistake in the defect detection process. To deal with the latter, we apply the matching algorithm to three consecutive frames. Using the matching results, we can interpolate the locations of the defects that are missed in the middle frame but are detected in the preceding and succeeding frames. Figure 15 illustrates the tracking results on three image sequences captured under different lighting conditions.

One natural extension from the foundation of an accurate director field is to identify the microtubule angles at the defect locations, which may provide insights into the dynamics of the active nematic system. This can be conveniently achieved using a similar CNN network introduced in Section 2.2 and Section 2.3. The image processing method (TM) can also derive approximate defect orientations (details are in Appendix A.1). To obtain accurate angles at defect locations, we manually labeled 800 defects using the Python package “PyQt5”, out of which, 390 were negative and 410 were positive defects. We ran our model and the TM on the same images and compared the angle differences with the ground-truth. Among these defects, our model and the TM resulted in average errors (defined as the difference in detected angle from the ground-truth angle) of $9.86^{\circ}$ and $14.33^{\circ}$ respectively. The comparison results are summarized in Figure 16.