ThermalNeRF: Thermal Radiance Fields

Thermal cameras detect and measure the heat signature of objects, and convert this emitted infrared (IR) energy into a thermal image reflecting varying levels of IR radiation [24]. Thermal images can provide insights into scenes and objects that are invisible to visible-light cameras [25]. The contactless nature of thermal imaging adds to its attractiveness in a diverse range of applications, including security and surveillance [11], preventive maintenance [20, 19], building inspection [26], monitoring rock masses [27], and archaeology [28, 29].

Thermal cameras that measure long-wave infrared (LWIR) wavelengths (8–14 $\mu$m) use very expensive germanium lenses that transmit in the IR spectrum but block visible light [30, 31]. The longer wavelength range also requires each element in the detector array to be larger than those required for the visible spectrum [32]. These factors contribute to the significantly lower spatial resolution [30, 33] and higher production cost [34] of thermal cameras compared to visible-light cameras. Considering this limited resolution, thermal cameras are often paired with other sensors to improve the effective resolution beyond what the thermal camera alone can provide [35, 36].

In particular, 3D scene reconstruction from IR images is a compelling method to achieve such sensor fusion and enhanced resolution, alongside a volumetric scene representation that is of independent value in many applications. Prior work has proposed methods for 3D IR reconstruction by combining short-wave or near IR and visible light images [37], but these methods yield limited-quality reconstructions and are presently limited to the short-wave IR wavelengths below 1 $\mu$m (see section 2.3), for which the resolution challenge is far less severe compared to the 8–14 $\mu$m LWIR range we consider. Other approaches have also been proposed for 3D thermal imaging, but these either rely on contact maps from functional grasping [38], rather than non-contact camera measurements, or study non-line-of-sight object localization rather than direct thermal imaging [39]. In contrast, our focus is to produce high-fidelity 3D thermal reconstructions of diverse scenes by combining measurements from a low-resolution LWIR camera and a higher-resolution RGB visible light camera, without any direct scene contact.

3D reconstruction is a valuable aid in the visualization, survey, and analysis of large or inaccessible objects or landscapes, in ways that 2D images are unable to provide [40, 12, 41]. Broadly, 3D reconstruction can be done based on Structure from Motion (SfM) algorithms using point clouds, or based on fully volumetric models such as radiance fields or signed distance fields.

Structure from Motion (SfM) algorithms, such as COLMAP [22], recover both camera poses and 3D points by matching corresponding feature points in 2D images from different viewpoints [42]. After camera poses are known, multi-view stereo (MVS) can then recover a denser 3D reconstruction by matching points across calibrated images [23]. Although MVS often works well, its quality is limited by the accuracy of the camera parameters computed by SfM, as well as reconstruction assumptions such as scene rigidity [43, 44]. In particular, COLMAP is used as a preprocessing step in nearly all radiance field models, as a way to recover camera parameters [23, 22]. It is usually successful on natural scenes with sufficient high-quality input images, but tends to fail on scenes with limited texture (yielding limited keypoints) or images of low-resolution [45].

Our work focuses on reconstructing a radiance field as a dense representation of a scene that leverages differentiable rendering instead of feature mapping. Many methods have been proposed for parameterizing a radiance field, including multilayer perceptrons [46, 47], voxel grids [48, 49, 50], factorized tensors [51, 52], multi-resolution hash tables [53], and anisotropic 3D points [54]; we build our method on top of one such representation, the nerfacto model provided by NeRFStudio [21]. While these methods represent great progress in producing faithful and high-resolution radiance fields, they are all focused on the visible spectrum for which high-resolution visible light camera images can be easily acquired. Extending radiance field reconstruction to represent other parts of the spectrum which are invisible to the human eye would expose useful and otherwise inaccessible information [55].

Capturing data beyond the visible spectrum can be helpful in identifying features that are invisible to the usual RGB color channels [56, 57, 58, 59, 60]. Accordingly, integrated sensors have been proposed to extend RGB cameras to incorporate information beyond the visible spectrum.

For instance, X-NeRF [37] reconstructs 3D from multispectral images by optimizing a transform between RGB and other sensors including near IR (NIR) cameras [61], while assuming known camera intrinsics. NIR images tend to have higher resolution as compared to long- and mid-wave IR (LWIR and MWIR) images due to its shorter wavelength [62, 63]. When applied to LWIR images, the novel views rendered from X-NeRF’s 3D multispectral reconstruction have relatively low-resolution, suggesting there is room for improved processing of longer wavelength IR images. Additionally, X-NeRF adopts the same assumption as vanilla NeRF: that each material absorbs different wavelengths of light equally, having a shared density parameter regardless of wavelength. While this assumption is a reasonable one for imaging a small range of wavelengths, it becomes problematic for sensor fusion across a wider spectrum like the one we consider.

In addition to these challenges inherent to multispectral imaging, our LWIR spectrum of interest poses additional challenges to 3D reconstruction. LWIR imaging faces an inherent physical resolution limitation due to its longer wavelengths, and consumer-grade thermal cameras like the handheld FLIR One Pro [36] that we use, often have even poorer image resolution [27, 64]. This poses multiple challenges to obtaining the camera poses and completing the subsequent 3D reconstruction [65]. We propose a method for thermal 3D reconstruction of scenes using both RGB and thermal images, leveraging cross-camera information to address these challenges [12]. While existing approaches have made use of similar insights in complementary tasks such as dehazing [66], hyperspectral imaging [67], and 3D reconstruction of a person via reflections [68], we demonstrate the first 3D RGB-thermal field reconstruction method that separately models material interactions with thermal and visible spectra to improve reconstruction quality.

Existing radiance field models, including NeRF [46] and its many variations, typically focus on modeling radiance in the visible spectrum as three color channels (red, green, and blue). In using the standard volume rendering formula based on the Beer–Lambert law (see section 3.2), these models implicitly assume that each point in space is equally absorptive of all three of these colors of light. While this is a good approximation for RGB visible light, to which most materials are either opaque or transparent, there are certain materials, such as stained glass, for which the approximation is no longer valid. For example, a red stained glass window transmits red light but occludes green and blue light, violating the equal-absorption assumption.

When we begin to consider radiance fields across a wider spectrum, including our setting of RGB and LWIR thermal radiance field modeling, we find that more materials exhibit differing absorption behavior across this wider spectrum. We model this behavior by explicitly endowing each spatial location with separate densities (absorption coefficients) for each wavelength, while introducing regularization to encourage these wavelength-specific densities to remain similar for most materials.

In the standard RGB setting, NeRF represents a scene as a volumetric radiance field

mapping a 3D point $\mathbf{x}\in\mathbb{R}^{3}$ and viewing direction $\mathbf{d}\in\mathbb{R}^{3}$ to a volume density $\sigma_{\text{rgb}}$ and view-dependent emitted color $\mathbf{c}_{\text{rgb}}=(r,g,b)\in\mathbb{R}^{3}$. The scene is rendered along a camera ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$ with origin $\mathbf{o}\in\mathbb{R}^{3}$ and direction $\mathbf{d}\in\mathbb{R}^{3}$ via standard volumetric rendering [69]

which is approximated numerically via $N$ samples along the ray via

Now, to extend NeRF from the RGB to the RGBT domain, given a ray $\mathbf{r}$, we wish to render the RGB color $(r,g,b)$ plus the color from a thermal image $\tau$. We treat this as 4-D color $\mathbf{c}=(r,g,b,\tau)$. Hence we introduce $c_{\text{therm}}:=\tau\in\mathbb{R}$:

We observe that in the visible-light spectrum, objects of interest tend to absorb wavelengths of light similarly, but the same is not true in the combined infrared-and-visible-light spectrum. For example, many objects are opaque to visible light but transparent to infrared light, or vice versa, as illustrated by the pyrex glass bowl in fig. 2, which is thermally opaque but visibly transparent. We therefore propose to render $c_{\text{therm}}(\mathbf{r})$ with a separate density $\sigma_{\text{therm}}$, distinct from $\sigma_{\text{rgb}}$:

We thus represent the RGBT scene as a radiance field

To optimize $F_{\Theta}$, we minimize the following objective:

Here $\mathcal{L}_{\text{rgb}}$ and $\mathcal{L}_{\text{therm}}$ are the standard pixel-wise photometric $\ell_{2}$ losses against calibrated ground-truth (gt) images:

where $\mathcal{R}_{\text{rgb}}$ and $\mathcal{R}_{\text{therm}}$ are rays from RGB and thermal cameras respectively.

$\mathcal{L}_{\text{$\sigma$}}$ is an $\ell_{1}$ regularizer encouraging the RGB and thermal densities to deviate from each other only at sparse 3D positions $\mathbf{x}$:

In practice, we implement this regularizer in two parts, where one part applies the regularizer with weight $\lambda_{\text{$\sigma$},\text{rgb}}$ to $\text{$\sigma$}_{\text{rgb}}$ (while blocking any gradients to $\text{$\sigma$}_{\text{therm}}$) and the other applies the same regularizer but with a larger weight $\lambda_{\text{$\sigma$},\text{therm}}$ to $\text{$\sigma$}_{\text{therm}}$ (while blocking any gradients to $\sigma_{\text{rgb}}$). This two-part implementation of $\mathcal{L}_{\text{$\sigma$}}$ allows us to prioritize transfer of information from the RGB to the thermal components of the reconstruction, while still allowing each spectrum to regularize the other. This two-part $\mathcal{L}_{\sigma}$ regularizer is motivated by the observation illustrated in fig. 2 that, while some materials (like the glass bowl) do absorb visible and infrared light differently, most materials are similarly absorptive of both visible and infrared wavelengths. For those objects, we can leverage the information in the higher-resolution RGB measurements to guide the thermal reconstruction and achieve thermal super-resolution.

$\mathcal{L}_{\text{cc}}$ is a variation on the cross-channel prior [70, 71] adapted to our radiance field reconstruction task:

where $\mathbf{c}^{\text{gt}}_{\text{rgb}}=(r^{\text{gt}},g^{\text{gt}},b^{\text{gt}})$. Concretely, we estimate $\mathcal{L}_{\text{cc}}$ stochastically by selecting a batch of patches of pixels during each minibatch gradient update. For each pixel patch, we convolve with a 2D finite difference kernel to compute the local spatial gradient in each channel, and then penalize the thermal channel for $\ell_{1}$ deviation in gradient relative to the visible channels. Note that $\mathcal{L}_{\text{cc}}$ is only applied to rendered thermal views for which we have ground truth RGB (rather than rendered RGB views for which we have ground truth thermal), so that this loss only affects the thermal reconstruction. Intuitively, this loss encourages object edges (high spatial gradient magnitudes) to align between the thermal and visible reconstructions. Since the RGB resolution of the training images is higher than the thermal resolution, this unidirectional cross-channel loss promotes thermal super-resolution, which demonstrate in fig. 13.

$\mathcal{L}_{\text{tv}}$ is a pixelwise total variation regularizer on thermally-unsupervised rendered thermal views:

Concretely, we estimate $\mathcal{L}_{\text{tv}}$ stochastically by selecting a batch of pixel patches during each minibatch gradient update and computing the 2D finite differences to estimate the local spatial gradient. We do this only for rendered thermal views for which we do not have ground truth thermal observations (i.e. views for which we only have RGB supervision). We motivate the inclusion of this term with two observations. First, thermal photographs tend to exhibit sparse features. Second, thermal cameras tend to have lower field of view (FOV) than RGB cameras [72]. Hence, especially when rendered from the perspective of an RGB camera, thermal views of the scene can exhibit noisy artifacts at the edges of the image or in the background. The inclusion of $\mathcal{L}_{\text{tv}}$ discourages the appearance of such artifacts and expands the boundaries of the thermal scene, compensating for the typically lower FOV of the thermal camera.

We calibrate both RGB and thermal cameras using OpenCV’s camera calibration [73] on a 2mm thick aluminum sheet with a 4 $\times$ 11 asymmetric grid of circular cutouts. Each circle has a diameter of $15$mm and a center-center distance of $38$mm. We chill the aluminum sheet in a freezer, then collect a series of simultaneous RGB and thermal calibration photographs. Example such photographs are shown in fig. 7.

For each of the RGB and thermal cameras, we estimate the extrinsic parameters, the rotation $R$ and translation $t$ from the camera coordinates to the world coordinates. The camera space coordinates $(X_{c},Y_{c},Z_{c})$ corresponding to a point $(X,Y,Z)$ in world space are

We additionally estimate the intrinsic parameters, the focal length $(f_{x},f_{y})$ and principal point $(c_{x},c_{y})$, as well as the radial distortion coefficients $k_{1},k_{2}$ and the tangential distortion coefficients $p_{1},p_{2}$ of each camera. Then a pixel $(u,v)$ is determined by

where

and

After using OpenCV’s camera calibration [73] to solve for the intrinsic and extrinsic parameters of both cameras, we compute the relative transform between the RGB camera and the thermal camera. We use COLMAP [22, 23] to estimate camera poses for our RGB camera for each scene, and then recover thermal camera poses using this calibrated relative pose. We resolve the global scale ambiguity in applying this camera offset by measuring the physical distance between two of our camera measurements for each scene.

We build and train our model by building upon the NeRFStudio library [21], using a modification of the default nerfacto model. Specifically, we create a second copy of the entire nerfacto model (including the auxiliary sampling model) to represent the thermal scene, and train the two models in parallel on their respective training images. The only connection between the RGB and LWIR (thermal) models is defined by our regularizers $\mathcal{L}_{\text{$\sigma$}}$ and $\mathcal{L}_{\text{cc}}$.

We test our method on a novel dataset of 9 real-world scenes, each with 50–170 images from distinct viewpoints. We collected these images using a handheld FLIR One Pro [35], which attaches to a smartphone and records simultaneous RGB and LWIR channel images at a resolution of 1080 $\times$ 1440 and 480 $\times$ 640, respectively. For each scene, we reserve 10% of the RGB-LWIR image pairs as a test set, and train on the remaining 90%. We selected these scenes to demonstrate a range of indoor and outdoor settings for which thermal imaging shows interesting phenomena, such as revealing heat sources and sinks, checking thermal insulation, and imaging through visibly occlusive media. Example images from each scene are shown in fig. 2, fig. 4, fig. 5, fig. 6, fig. 8, fig. 9, fig. 10, fig. 14, and fig. 16 in the supplement.

In addition to this real-scene dataset, we also introduce a synthetic RGBT scene based on the hotdog scene from the NeRF Blender dataset [46], in which we make the hotdog thermally hot. This synthetic scene allows us to separately test the regularization and modeling aspects of our approach, without any concerns over potential miscalibration. It also allows us to test our thermal super-resolution capability in a setting for which ground-truth high-resolution thermal images are available, which is not the case for our real-scene dataset due to the limited thermal resolution of the FLIR One Pro. For this synthetic dataset, we render 45 training views and 5 testing views, each with red, green, blue, and thermal channels.

We show qualitative results on all of our real-world scenes in fig. 2, fig. 4, fig. 5, fig. 6, fig. 8, fig. 9, fig. 10, fig. 14, and fig. 16 in the supplement. We note that, while all methods tend to do decently well at reconstructing the RGB scene, only our method reliably also recovers the thermal scene. Treating the thermal and visible spectra separately ({RGB}{T}) typically results in catastrophic failure to reconstruct the thermal scene, because the low resolution of the thermal images causes the camera pose refinement of NeRFStudio [21] to diverge—a similar behavior as in fig. 3. Treating the thermal and visible spectra jointly ({RGBT}), with a shared density field, is successful on many objects but fails to reconstruct certain materials, like glass, and often exhibits unintended leakage from the visible to the thermal spectra.

In Table I we compare our method to two natural baseline approaches based on nerfacto [21]. The {RGB}{T} baseline consists of completely separate nerfacto models for the visible and thermal wavelengths, while the {RGBT} baseline consists of a single nerfacto model with an extra color channel to represent the thermal (LWIR) spectrum. Our method is similar to the {RGB}{T} baseline, but with regularizers to tie together the visible and thermal reconstructions for most materials.

While both of these baselines successfully recover the RGB scene–unsurprisingly since the nerfacto model was developed for RGB imaging–only our method is also successful at simultaneous thermal reconstruction. The {RGB}{T} baseline tends to fail dramatically due to diverging thermal camera pose refinement (built into NeRFStudio [21]), while the {RGBT} baseline suffers thermal artifacts for not allowing any density (absorption coefficient) variation between the visible and thermal reconstructions.

Table I also includes ablation studies comparing our method to versions of it without each of our three regularizers, $\mathcal{L}_{\text{cc}}$, $\mathcal{L}_{\text{$\sigma$}}$, and $\mathcal{L}_{\text{tv}}$. Note that our method without any regularization is identical to the {RGB}{T} baseline, except that we also reduce the degree of camera pose refinement.

We find that including any regularizer alone produces a substantial improvement in thermal reconstruction quality relative to the two nerfacto baselines, with modest improvement in RGB reconstruction quality. We note that, in addition to the modest quantitative impact of $\mathcal{L}_{\text{$\sigma$}}$ shown in table I, this regularizer produces meaningful qualitative improvement. This qualitative improvement can be seen via the RGB and thermal depth maps in fig. 11, and by enabling the de-occlusion application shown in fig. 12.

In fig. 12 we demonstrate an application made possible by our method, specifically by $\mathcal{L}_{\text{$\sigma$}}$ and the use of separate densities for visible and LWIR light. We can remove occluding objects from RGB or thermal views, thus revealing objects hidden behind other objects by rendering only the parts of the scene with RGB and thermal densities sufficiently similar to each other. Precisely, we render RGB and thermal scenes respectively with densities

where $\epsilon$ is the minimum allowed magnitude of difference between the RGB and thermal densities in the rendered image. This computation is only possible due to the $\lambda_{\text{$\sigma$}}\mathcal{L}_{\text{$\sigma$}}$ term in our loss function (eq. 10), which encourages the RGB and thermal densities to be similar.

We demonstrate this application on the pyrex scene, which depicts a cold pack in a glass container, and the sheet scene, which depicts a hot water kettle behind a plastic sheet. The glass container in pyrex transmits visible but not infrared light; the plastic sheet in sheet transmits infrared but not visible light. We show that we are able to remove the occluding material in the RGB and thermal channels respectively, revealing the shape of the object that normally would be occluded from the ground-truth view.

In fig. 13, we show qualitatively that our method achieves a level of thermal super-resolution. We demonstrate super-resolution on our synthetic hotdog scene, for which ground truth thermal images are available at high resolution for a valid comparison. We show that with the help of the higher-resolution RGB images (column 1), despite the low resolution of the thermal training data (column 2), our reconstruction (column 4), is able to reconstruct the higher-frequency ground-truth thermal features (column 3) in the underlying volume.