Spinning Metasurface Stack for Spectro-polarimetric Thermal Imaging

The demand for high-resolution, information-rich image data has been amplified by the widespread industry adoption of machine learning algorithms[1, 2, 3, 4, 5, 6]. Meta-optics-enabled spectral and polarimetric imaging exhibits potential in meeting these rising data demands of learning algorithms within the visible spectrum[7, 8, 9, 10, 11, 12, 13, 14, 15]. Yet, the integration of meta-optics with infrared thermal imaging remains a relatively unexplored domain. Recent strides in heat-assisted detection and ranging (HADAR) have demonstrated the potential of thermal imaging[16] for machine perception tasks utilizing the infrared spectrum and the atmospheric transparency window. By combining spectral-resolved thermal imaging with artificial intelligence, HADAR offers a platform for machine perception through pitch darkness like broad daylight [16]. Here, our goal is to show that integrating meta-optics with infrared thermal imaging emerges as a crucial enabling factor for such next generation machine vision algorithms. Our work offers a pathway for realizing compact spectro-polarimetric thermal imagers beyond the conventional technologies which use push-broom, filter-wheel or interferometer modules. This new class of thermal imagers enhances the ability of machine learning algorithms for capturing information such as temperature, material composition, and surface morphology.

Within the broad thermal infrared spectrum, the long-wave infrared (LWIR) region stands out as especially advantageous for heat-assisted detection and ranging, as most objects at room temperature radiate thermal energy at these wavelengths (Fig. 1a). Moreover, the LWIR atmospheric transmission window facilitates the thermal radiation signal to propagate long distances[17]. However, the blurry nature of thermal images poses a significant challenge for machine vision algorithms. For example, perception tasks such as semantic segmentation and ranging rely on texture and contrast for learning features and show poor performance with panchromatic thermal imagery[18]. HADAR overcomes the blurry nature of thermal images by decomposing the LWIR thermal radiation into its spectral channels[16]. Unfortunately, the commonly used mosaic filter technique for spectral decomposition (Fig. 1b)[19, 20, 21, 22, 23, 24] falls short for LWIR thermal imaging due to the restricted pixel count in LWIR focal plane arrays. Current HADAR spectral imagers rely on cumbersome filter wheels (Fig. 1c) with restricted spectral resolution, or fragile and bulky interferometers (Fig. 1d) with a confined field of view (FOV) [16]. Such constraints considerably hinder the widespread applicability and adoption of HADAR.

To address these limitations, in this work, we harness the capabilities of large-area meta-optics in the infrared domain to create a platform for spectro-polarimetric thermal imaging. We achieve spectral decomposition by designing the dispersion and polarization through stacked metasurfaces, while the spectral reconstruction is accomplished using compressive sensing and dictionary learning algorithms. Our designs employ 2D structures with large feature sizes (> 1$\mu m$), which can be made using photo-lithography techniques, facilitating large-area fabrication and scalable manufacturing. Despite its compact form, our Spinning MetaCam delivers exceptional spectral resolution across diverse materials and can effectively unveil subtle polarimetric feature, enabling applications such as material classification . By leveraging meta-optics and advanced computational spectroscopy methods, we demonstrate three times higher accuracy for machine vision tasks than conventional panchromatic thermal imaging. In the future, the angular speed of the spinning stage can be increased to enable real-time video leading to widespread adoption of HADAR technology similar to RADAR, LIDAR, and SONAR.

The architecture of our spinning-metasurface based spectro-polarimetric imaging system is depicted in Fig. 1e. It comprises of a broadband linear polarizer, three anisotropic and dispersive metasurfaces, and an LWIR imaging sensor. The polarizer is utilized to polarize the incoming thermal radiation signals, and the metasurfaces are utilized to realize spectral filtering. We design the metasurfaces with high anisotropy to produce distinct spectral responses for orthogonal polarizations. Additionally, the metasurfaces’ dispersion rotates different wavelengths of radiation to varying polarization orientations. By using the metasurfaces in tandem and axially spinning the polarizer and metasurfaces to different angles, we obtain tunable transmission spectra that sample the incident thermal radiation in its spectral and polarimetric channels. We then reconstruct unknown spectra of imaging targets using compressive sensing and dictionary learning algorithms. Dictionary learning generates a set of basis functions that represent the unknown spectra in a sparse format [25]. Compressive sensing enables accurate reconstruction of the sparse spectra from limited number of measurements [26]. Combining these two techniques enables accurate and stable spectral reconstruction in the presence of noise and measurement errors[27]. The four-dimensional spectro-polarimetric data generated by our system offers a wealth of physical information about an imaging target, making it a valuable tool for physics-driven machine vision[5, 28], facilitating various tasks such as object detection and semantic segmentation[29, 30].

To quantitatively describe the mechanism of the spinning-angle-controlled transmission spectra, we represent the spectro-polarimetric response of the metasurfaces using Jones matrices. Assuming that the transmission axis of the input linear polarizer is at 0 degrees relative to x-axis, the Jones matrix $J_{i}$ of a metasurface i with the principal axis (p) at a spinning angle $\theta_{i}$ relative to the x-axis can be expressed as:

where R is the rotation matrix, $J_{Mi}$ contains the anisotropic transmission of the metasurface $t_{ip}$ and $t_{is}$ along the two principle axes p and s.

The Jones matrix of the three-metasurface assembly is given by:

Thus, the total transmission spectrum of the three spinning metasurfaces strongly depends on the spinning-angle combinations $\Theta=(\theta_{1}$, $\theta_{2}$, $\theta_{3})$ when the constituted metasurfaces are strongly anisotropic and dispersive, i.e. $t_{ip}(\lambda)\neq t_{is}(\lambda)$ (see Supplemental Document for the detailed analysis). We note that large differences between the spectral responses of the three metasurfaces (M1, M2, M3) are also introduced to minimize the correlations between the generated spectra, which can significantly improve the spectral reconstruction performance [31]. We note that the metasurfaces do not necessitate rotation at fixed angular frequencies during the imaging process. As depicted in Eq. 2, it is the static angular positioning of the metasurfaces that is pivotal for selectively filtering the incident thermal radiation in its spectral content. We emphasize that our design generates a large set of distinct transmission spectra with only three metasurfaces, while the total number of spectra in traditional mosaic array is limited to the number of metasurfaces/filters used [19, 20, 21, 22, 7, 8, 23].

Accordingly, we design the metasurfaces and experimentally achieve three key characteristics for optimized spectro-polarimetric imaging performance: 1) Strong anisotropy and dispersion for efficient wavelength decomposition; 2) High transmission and low self-emission for high signal to noise ratio (SNR); 3) Small angular dependence for a large FOV. The unit cell of the three designed metasurfaces are shown in Fig. 2 a-c. Strong dispersive anisotropy of the transmission spectra can be observed in Fig. 2e. Additionally, we emphasize that large-area devices are generally required for imaging applications to ensure sufficient numerical aperture. All the metasurfaces designed here have feature sizes larger than 1 $\mu m$. Large-area devices (25.4mm in diameter) with high structural quality and uniformity (Fig. 2d) can be rapidly fabricated by standard photo-lithography techniques, enabling scalable manufacturing for practical applications. This is in strong contrast to recent works on miniaturized spectrometers[32, 33, 34, 35, 36, 37, 38, 39, 40], where the device footprint is on the micrometer scale and thus not suitable for imaging applications.

The tunable transmission spectra produced by our spinning metasurfaces are shown in Fig. 2f. The distinct spectra are a result of the tuned spinning-angle combination $\Theta$. We integrate the three fabricated metasurfaces tandemly via compact rotatory mounts to independently control the rotation of each metasurface (Inset of Fig. 2f). We also optimize the spinning-angle combinations of the three spinning metasurfaces using genetic algorithms to generate largely uncorrelated transmission spectra for optimal spectral reconstruction performance (see Supplemental Document for details). Additionally, we note that increasing the number of metasurfaces can further improve the spectral resolution, but simultaneously reduces the SNR as the peak transmissions of the LWIR devices are limited to around 0.6 (see Supplemental Document figure S5 for details). However, our method has the potential to scale up into the hyperspectral regime by adding more high-transmission LWIR metasurfaces.

We also evaluate the FOV of our imaging module by integrating it with an LWIR thermal camera and capturing images of a large area uniform blackbody. To determine the spatial transmission efficiency, we normalize the signal counts of each pixel by the counts at the center of the images. We also define the angular range with transmittance above 0.5 as the effective FOV of a system. As seen in Fig. 2g, our spinning metasurface module has an FOV of around 20.5 degrees, which is difficult to achieve with interferometer-based spectral imagers.

To extract the unknown spectro-polarimetric properties of various imaging targets, we use a combination of dictionary learning and compressive sensing algorithms in the reconstruction process. The tunable transmission spectra produced by our spinning metasurfaces (shown in Fig. 2f) are not narrowband, which means that the collected raw signals at different spinning-angle combinations $\Theta$ do not directly reflect the spectral radiance at different wavelengths. Instead, the collected signal $I(\Theta)$ at each pixel can be described as an integral of the spectral response function $R(\Theta,\lambda)$ multiplied by the ground truth spectrum $P(\lambda)$ that we wish to obtain, i.e. $I(\Theta)=\int_{\lambda_{\min}}^{\lambda_{\max}}R(\Theta,\lambda)P(\lambda)\mathrm{d}\lambda$. To solve for this equation, we discretize the spectral range of interest and express it in a tensor form as shown in Eq. 3:

We emphasize that directly solving Eq. 3 does not produce accurate spectral reconstructions[41, 33]. In theory, we can use measured signals $I_{\Theta}$ and the pre-calibrated response function $R_{\theta\lambda}$ to directly determine unknown spectra $P_{\lambda}$ at each pixel of a scene. However, in practice, two limitations impede the performance of spectral reconstruction: the problem becomes underdetermined when there are many discretized wavelength bands, and measurement noise affects both $I_{\Theta}$ and $R_{\Theta\lambda}$, making the direct reconstruction method unstable and the results inaccurate.

To enhance both the precision and consistency of spectral reconstruction, we leverage the capabilities of compressive sensing and dictionary learning algorithms to solve Eq. 3 (Fig. 3). Our approach begins with dictionary learning, a process we employ to generate a dictionary comprised of 32 basis functions, designated as $D$. The spectra utilized for the dictionary learning are derived from the infrared emissivity spectra drawn from the ECOSTRESS Spectral Library [42, 43], as well as blackbody thermal radiation spectra across various temperatures. The resultant dictionary can provide sparse representations for any thermal radiation spectrum in the space of spectra we are studying. We point out the efficiency of this sparse coding is fundamentally linked to the prevalent spectral similarities found within the LWIR thermal radiation spectra. We thus project the unknown spectrum $P_{\lambda}$ as a linear combination of the basis functions in the dictionary (Fig. 3 c and d). We have,

where $\phi_{k}$ is a sparse coding of the spectrum $P_{\lambda}$. With this sparse representation, the spectral reconstruction problem can be solved by first obtaining $\phi_{\text{recon}}$:

where $A_{\Theta k}=R_{\Theta\lambda}D_{\lambda k}$, and $\epsilon$ is the residual error. Finally, the spectra $P_{\lambda}$ at each pixel of a scene is reconstructed by

Our reconstruction method significantly improves the reconstruction accuracy, making the spinning-metasurface-based spectro-polarimetric imaging more robust against noise and measurement errors.

To evaluate the performance of our prototype imaging system, we conduct experiments using a custom-designed "PURDUE" target made of letters constructed from titanium and a glass substrate (Fig. 4a). Each letter has unique micro-grating structures (Inset of Fig. 4a) that generate distinctive spectro-polarimetric signatures in the thermal radiation signal. The glass substrate also features a characteristic emission peak around 11 $\mu m$. Note that we heat the image target to 150${}^{\circ}C$ to generate high signal intensity. The reconstructed spectra of four representative pixels are shown in Fig. 4 b-e. We compare them with the ground truth spectra measured by a Fourier-transform infrared spectrometer, validating the effectiveness of our reconstruction approach. The reconstructed spectral frames (Fig. 4f) also exhibit high contrasts between different wavelengths, demonstrating that the system can effectively reveal the LWIR spectral properties of different targets. We note that the relatively low reconstruction accuracy at shorter wavelengths (8 - 10 $\mu m$) results from the low transmission (low SNR) and the high correlation (similarity) between the tuned spectra (Fig. 2f).

To characterize the spectral resolution of our system, we conduct a numerical simulation of the spectral reconstruction performance using Gaussian peaks centered at varying wavelengths. This testing method is a widely adopted way to determine the fundamental resolution constraints of reconstruction-based spectroscopy [21, 22, 24, 33, 35, 41]. The simulation incorporates the tunable transmission spectrum set (as depicted in Fig. 2f) produced by the spinning-metasurface module, and utilizes the same dictionary learning and compressive sensing algorithms previously discussed. Fig. 4i reveals that Gaussian peaks with a 0.6 µm full width at half maximum (FWHM) centered above 10 µm can be precisely reconstructed with negligible mean square errors (MSE). Additional simulations also show that our system is capable of detecting narrower spectral peaks with FWHM as low as 0.1 micron (see Supplemental Document figure S4 for details). We again observe that the performance of spectral reconstruction is dependent on wavelength. A higher level of accuracy is achieved at longer wavelengths due to the significant differences between the tuned transmission spectra in this region (Fig. 2f). This implies that we could further enhance the spectral resolving power by refining the design of the metasurfaces and generate a broader array of mutually uncorrelated tunable transmission spectra.

We also obtain the polarimetric information including degree of the linear polarization (DOLP) and the angle of linear polarization (AoLP) using the designed system. For polarimetric imaging, we collectively rotate the spinning metasurfaces and the input polarizer, selecting four different polarizations ($0^{\circ}$, $90^{\circ}$, $45^{\circ}$ and $-45^{\circ}$) while maintaining the same spectral transmission. We use the first three Stokes parameters to quantify the polarimetric information associated with each pixel, i.e. $S_{0}=I_{0}+I_{90}$, $S_{1}=I_{0}-I_{90}$, and $S_{2}=I_{45}-I_{-45}$, where $I_{0}$, $I_{90}$, $I_{45}$ and $I_{-45}$ are the light intensity at polarization angles of $0^{\circ}$, $90^{\circ}$, $45^{\circ}$ and $-45^{\circ}$, respectively. The DoLP and AoLP are then calculated at each wavelength through $\mathrm{DoLP}=\sqrt{S_{1}^{2}+S_{2}^{2}}/S_{0}$ and $\mathrm{AoLP}=arctan(S_{2}/S_{1})$. As shown in Fig. 4 g and h, we can clearly distinguish between different letters based on their polarimetric signatures in the thermal radiation signal. The four-dimensional spatial-spectro-polarimetric data-tesseract provides significantly more insight associated with an object, making it a powerful tool for a wide range of imaging applications.

To prove the effectiveness and reveal the potential of Spinning MetaCam, we conduct numerical tests demonstrating its applicability in material classification. Fig. 5a shows that our spinning metasurfaces can accurately reconstruct the LWIR thermal radiation spectra of a diverse range of materials commonly found in everyday life (ground truth obtained from the ECOSTRESS Spectral Library [42, 43]). These results also highlight the spectral fidelity and universality of our spectral reconstruction method. Employing these thermal radiation spectra, we execute material classification by identifying the material whose ground truth spectrum exhibits the least mean square error (MSE) when compared to the reconstructed spectrum. We repeat this spectral reconstruction and material detection procedure 100 times for each material, and use the outcomes to generate a confusion matrix (see Supplemental Document for details). Fig. 5b and Fig. 5c compare the resulting confusion matrices derived with and without the use of spinning metasurfaces, while maintaining a consistent noise level of 5%. Without the spinning metasurfaces, material classification relies solely on the integrated intensity of the thermal radiation over the LWIR region. This approach exhibits a low precision for most materials, being effective only for those materials with substantially disparate thermal radiation intensities, such as metallic paint. In stark contrast, material classification significantly improves in accuracy when the spectra collected by the Spinning MetaCam is utilized. We note that instances of lower accuracy only arise when there is a pronounced similarity in the LWIR spectra of materials, as observed between construction glass and felsic. For a more quantitative analysis, we perform the material classification test at varying noise levels ranging from 1% to 10%, and subsequently calculate the mean average precision (mAP)[44]. Representing the overall accuracy of classification for all classes of materials at all different noise levels, the mAP reaches an impressively high value of 0.834, providing a significant improvement from the low mAP of 0.293 obtained without the spinning metasurfaces.

Our results provide an innovative approach for spectro-polarimetric thermal imaging by combining meta-optics and computational spectral reconstruction. The low-SWaP (size, weight, and power) system opens the door for physics-driven machine vision. The high-dimensional thermal image data can significantly improve the performance of tasks such as depth estimation, object detection, and semantic segmentation when only radiative heat signal is available. Furthermore, we foresee that spectro-polarimetric thermal imaging can also be a powerful tool for scientific research, allowing for non-destructive characterization in the infrared region to investigate a wide range of novel physical phenomena, such as anisotropic thermal conduction[45] and directional or nonreciprocal radiative heat transfer [46, 47, 48]. Overall, our work provides a key development in the rapidly growing field of thermal imaging, offering a pivotal technology for heat-assisted detection and ranging.

Funding This work was supported by the United States Department of Energy, Office of Basic Sciences under DE-SC0017717, and the Defence Advanced Research Projects Agency (DARPA) under the Nascent Light-Matter Interactions (NLM) program.

Disclosures The authors declare no conflicts of interest.

Data availability Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document See Supplemental Document for supporting content.