Photometric Correction for Infrared Sensors

An infrared camera is a device that converts infrared radiation into a visual image that depicts temperature variations across an object or scene. Infrared radiation is a characteristic of all objects that have a temperature higher than absolute zero (zero Kelvin or -273 Celsius) [29]. Thermal energy radiated by scene objects is focused onto the sensor image plane where a grid of microbolometer sensors is placed to convert the optical energy focused onto each grid element into a pixel voltage indicative of the object temperature.

The physical response of each pixel element is governed by a heating and cooling mechanism as the camera shutter is opened and closed. Similar to RGB optical sensors, microbolometer sensors integrate incident radiation into stored charge when the camera shutter is open and dissipate the stored charge when the camera shutter is closed. This process generates a response analogous to an RC circuit driven by a square wave excitation where the square wave period is determined by the exposure time and its amplitude is determined by the radiated energy of the scene onto the pixel sensor. The rate of integration and dissipation is driven by a sensor-specific time constant, $\tau$.

One shortcoming of microbolometer sensors is their response time. RGB pixel sensors based on CMOS technology have been shown to have time constants $\tau\in[1\mu s,500\mu s]$ with the median sensor performance across RGB sensors $\tau\approx 10\mu s$ at room temperature [5]. Typical response times for microbolometer sensors are $\tau\in[8ms,15ms]$ which is more than two orders of magnitude larger. Long response times lead to “ghosting” of sensed values where the value of a pixel in a past frame persists in the current frame creating a spatio-temporal blur of moving objects in sensed IR images.

Structure from Motion is the process of reconstructing a 3D structure from its projections into a series of images taken from different viewpoints. It commonly starts with feature extraction and matching, followed by geometric verification [25], where the feature matching searches for feature correspondences by finding the most similar feature between image pairs with scene overlapping and comparing the similarities between features. This requires the detected feature to be accurate and reliable throughout the image sequence. We note work in the literature on the related problem of visual odometry from IR image data which estimates the “M” or camera motion estimation component of the SfM problem [22, 17, 18]. Work in [8] demonstrates that RGB SfM algorithms may be used on IR cameras for use in firefighting scenarios with some success.

The photometric correction has proven to be important for improving the result of SfM estimates [9, 13, 11]. Direct Sparse Odometry (DSO) [9] achieves state-of-the-art performance by taking full advantage of photometric camera calibration, including lens attenuation, gamma correction [12], and brightness correction. The lens attenuation and gamma correction both require prior knowledge about the sensor and lens being used for solving SfM problems, the brightness correction, instead, is an empirical model that takes into account automatic exposure changes and compensates the pixel values in order to make them more consistent and stable across a sequence of images. The brightness correction function in DSO is given by $e^{-a}(I-b)$ where $a$, $b$ are the correction parameters, and $I$ is the pixel value. In their work, the authors show that a naive brightness constancy assumption used in other approaches like LSD-SLAM [10] or SVO [14] significantly decreases the SfM accuracy. This article builds on these concepts by translating this intuition from CMOS-based RGB image sensors to microbolometer-based IR image sensors. A photometric correction model for microbolometer pixel values is proposed in order to improve the performance for SfM problems using IR data.

Heating the IR pixel is modeled as a differential equation with a unit step forcing function, $\mu(t)$, where the amplitude of the step is proportional to the scene irradiance. We then seek to calculate the steady pixel value that reflects the unknown irradiance of the scene point which corresponds to the asymptotic of the step response. The rate of the convergence of the pixel value to the steady state response is determined by the heating time constant, $\tau_{h}$. The process of pixel heating up can be depicted in Fig. 1.

The model denotes the initial measurement time as $t=t_{0}$ and uses the “black body” assumption to set the initial value of the IR pixel, i.e., $I_{m}(t_{0})=0$.

A first-order differential equation model is provided in Eq. 1 to characterize heating an IR pixel.

where $I_{m}(t)$ is the pixel intensity value measured by the camera sensor at time $t$, and $I_{ss}$ is the steady state pixel intensity value if the microbolometers were given sufficient heating time.

Let $t_{e}=t_{1}-t_{0}$ denote the exposure time, meaning that the shutter is open from the beginning time $t_{0}$ to time $t_{1}$. Solving the first-order differential equation at $t=t_{1}$ gives

By modeling the heating process of a microbolometer, we find the value of the forced response at a steady state due to the excitation of the pixel from the associated portion of the 3D scene.

Cooling the IR pixel is modeled as the natural response of the same differential equation after the heating forcing function, $\mu(t)$, has been set to zero. We then seek to calculate the measured pixel value at the time $t=t_{0}+T$ where $T$ is the time of each frame. This pixel value will then contribute as a non-zero initial condition for the subsequent image at time $t_{0}+T$. The decay rate of the pixel value is determined by the cooling time constant, $\tau_{c}$. The process of pixel cooling is depicted in Fig. 1.

The model denotes the excitation of the pixel at the time that the forcing function is removed as $t=t_{1}$ when the shutter is closed, and uses the value of the measured pixel at $t=t_{1}$ as the initial value of the IR pixel, i.e., $I_{m}(t_{1})=I_{0}.$

Similarly, another first-order approximation is used to describe the cooling process.

Let $t_{r}=t_{0}+T-t_{1}$ denote the sensor readout time when the shutter is closed. Solving this first-order differential equation at time $t=t_{0}+T$ gives

By modeling the cooling process of a microbolometer, we find the pixel value at the end of each frame period, which is also the initial measurement for the next frame.

We consider sensors that record images at a framerate of $f_{s}$ or equivalently having a temporal sample period $T=\frac{1}{f_{s}}$. The time interval between each frame, $T$, is further subdivided into a measurement or exposure time during which time the shutter is open, $t_{e}$, and a readout time during which the shutter is closed, $t_{r}$. During the exposure time period, the microbolometer is heated. During the period that the sensor reads out the pixel values, the microbolometer is cooling. Fig. 1 shows the pixel value convergence when the microbolometer is heating and the heat residual from the previous frames left on the new frame when the microbolometer is cooling. We seek to calculate the steady state pixel value excited by a scene point with the prior heat residual removed. Assuming that the effect from previous frames is dominated by the most recent prior frame when there are no drastic temperature gradients in the scenes, the earlier frames are ignored in the model. The complete video sensing model in Eq. 5 merges these two models into a comprehensive model for the pixel response, $I_{i}^{\prime}(t_{e},t_{r})$ at frame $i$ while recording a time-sequence of images.

where $I_{i-1}$ and $I_{i}$ are two continuous frame, $t_{r,i-1}$ is the readout time in the previous frame $i-1$, and $t_{e,i}$ is the exposure time of the current frame $i$.

By combining the heating model and cooling model, the irradiance, or the temperature of the measured scene point, can be more accurately reflected by the stable pixel value $I^{\prime}_{i}$.

In the remainder of this paper, $I_{i}$ will always refer to the photometrically corrected image $I^{\prime}_{i}$, except where otherwise stated.

In DSO the authors develop advanced photometric correction models for both camera lens and RGB pixel sensing compensation. This is coupled with camera calibration data to perform highly-accurate SfM at real-time rates with impressive 3D structure reconstruction results. To leverage such a system and apply it to infrared sensors, the brightness transfer model used for RGB sensors in DSO is replaced by our infrared sensor photometric correction model with the following modifications in the SfM algorithm.

• •

  Added a time history (previous sensed values) to tracked pixels.

• •

  Modified the optimization approach to use the derivatives and Hessian of our photometric correction model.

To reconstruct a 3D scene from infrared images using structure from motion, a map is computed to associate two pixels in different frames that both correspond to the same 3D scene point. The photometric error between them is defined in a similar way as [9]. For a point, $\mathbf{p}$ in reference frame $I_{i}$, observed in target frame $I_{j}$, the photometric error, given by Eq. 6, is formulated as the weighted Sum of Squared Differences (SSD) over a small neighborhood of pixels.

where $\mathcal{N}_{\mathbf{p}}$ is the set of pixels in the SSD, and $\|\cdot\|_{\gamma}$ is the Huber norm. In addition to using robust Huber penalties, a gradient-dependent weighting $w_{\mathbf{p}}$ [9] is applied.

To minimize the photometric error between the corresponding points in two frames, the optimizer then optimizes the heating and cooling time constants $\tau_{c}$ and $\tau_{h}$, instead of the brightness transfer variables in DSO. The optimization is accomplished using the Gauss-Newton algorithm in a sliding window [21].

The FLIR ADAS Dataset consists of a video sequence of images taken from an IR camera and an RGB camera mounted to the front of a vehicle. The dataset was acquired with an RGB and IR camera mounted on a vehicle (car) where the IR sensor was a Teledyne FLIR Tau 2 thermal camera and the RGB a Teledyne FLIR Blackfly camera. Both RGB and IR videos were recorded at 30 frames per second (fps) under generally clear conditions during the night. Experiments use a subset of the complete ADAS dataset that corresponds to a video sequence of 600 images where the vehicle drives straight down a road at night. Example images from this video sequence are shown in Fig. 2. The results are summarized in two experiments:

• •

  Evaluations on the reconstruction quality of the road show that our photometric correction enables DSO to track more points on IR data and improve the accuracy of reconstruction.

• •

  Evaluations on the camera motion show that with the proposed correction model, the trajectory is more stable and less deviated in terms of being at a certain distance away from the RGB camera trajectory.

To show that the proposed photometric correction model can also be applied to other IR image-processing contexts, the proposed photometric correction model is applied to the BU-TIV dataset for solving a human being tracking problem. The BU-TIV dataset is designated for the object-tracking problem using infrared data and consists of video sequences in different scenes that were recorded with FLIR SC8000 cameras. A subset of the dataset of a daytime crowded street view during a marathon competition was used. The results are summarized in two experiments: (1) Experiment 1 tracks pedestrians in IR image sequences. (2) Experiment 2 tracks the observed temperature for a stationary target in the IR video.

Experiment 1 evaluates the proposed photometric correction for pedestrian tracking from an IR image sequence. Results are shown in Fig. 6 for three distinct photometric correction approaches: (1) RGB correction (RGB), (2) no correction (IR), and (3) the proposed correction (IR+cor). Table 3 shows the quality of each estimate as measured by three distinct metrics: (1) the Discrete Frechet (DF) distance [7] , (2) the Dynamic Time Warping (DTW) [3], and (3) a custom metric referred to as the Mean Distance. Mean distance calculates the average distance between the person’s estimated and actual positions for all corresponding frames. Table 3 indicates that the proposed IR correction improves the performance across all three metrics, where lower scores are better. The last row shows the number of frames where the person is tracked and again the proposed IR correction algorithm outperforms the other cases.

Experiment 2 tracks the temperature of a parked car roof as shown in Fig. 6 and seeks to analyze the stability of the temperature before and after applying the proposed IR photometric correction. Table 4 shows that the proposed photometric correction improves the stability of pixel intensity and, by extension, the estimate of the unknown constant temperature of the observed object.