In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking

Deep learning has significantly transformed the landscape of visual object tracking, introducing unparalleled accuracy and robustness [38]. This paradigm shift has catalyzed the development of sophisticated models capable of handling diverse and complex tracking scenarios. The integration of deep learning in object tracking began with works like DLT [39] and GOTURN [40], which demonstrated the efficacy of CNNs. This marked a departure from traditional correlation filter-based tracking methods [19, 41, 42, 43], paving the way for more advanced neural network applications. Subsequent advancements saw the emergence of various methods such as SiamFC [17], MDNet [37], SiamRPN [8], and SiamEXTR [12]. Each method offered unique strengths; for instance, Siamese networks gained popularity for their balance between accuracy and speed in real-time tracking applications [16]. The development and refinement of these methods have been greatly influenced by large-scale training datasets [44, 45, 46], and challenging benchmarks such as OTB [47], GOT-10k [48], and LaSOT [49]. These resources have been crucial in evaluating the performance and robustness of tracking methods under varied conditions. Despite advancements, challenges such as OCC, SV, and BC remain persistent issues in deep learning-based trackers. Moreover, the generalizability of these methods across different environments and objects is an area of ongoing research. Recent notable studies introduce the famous Transformer [50] architecture to the visual tracking community, achieving state-of-the-art performance in most benchmark datasets [51, 52, 53]. Besides, in the broader context of visual object tracking, significant advancements have been made by segmentation-based methods [54, 55, 56]. While these methods focus on visible data, their underlying principles offer valuable insights into feature extraction, semantic segmentation, and integrating diverse data types. These studies reflect the continuous evolution and refinement of visual tracking methods.

TIR object tracking, leveraging TIR imaging to detect and follow objects, plays a critical role in scenarios with limited or no visible light. This tracking modality offers unique advantages over traditional visual tracking, particularly in challenging visibility conditions [2]. The integration of deep learning into TIR tracking has marked a significant advancement [57]. Notable models such as CNNs and recurrent neural networks (RNNs) [58] have been adapted to enhance the accuracy and reliability of TIR tracking methods. Integrating infrared information with other modalities, like RGB data, has been explored to improve robustness. This multimodal approach, as demonstrated in MMMPT [59], shows promise in creating more resilient tracking systems under varying conditions. Significant recent contributions include MLSSNet [22], achieving breakthrough performance in distinguishing distractors, and MMNet [24], which innovated in learning dual-level deep representation. These studies showcase the rapid progress in TIR tracking. Emerging trends include the use of powerful Transformer architecture to further refine TIR tracking and the exploration of new feature-fusion technologies [60, 61, 62]. It is also noteworthy that segmentation-based tracking methods have not yet been developed for TIR object tracking due to the lack of accurate pixel-level annotations. We believe TIR tracking has undergone substantial growth, primarily driven by advancements in deep learning. The ongoing research in this field is poised to significantly enhance capabilities in complex scenarios, expanding its applicability across various domains.

Bayesian filtering has become a cornerstone technique in many visual tasks, offering a probabilistic approach to understanding and predicting dynamic systems [63, 64]. Its evolution from simple Bayesian methods to sophisticated filtering algorithms parallels the advancements in visual processing and computational power. At its core, Bayesian filtering involves updating beliefs over time using Bayes theorem [65]. Approaches like the Kalman filter [66] and particle filter [67] have been pivotal in visual tasks, each with its strengths in handling linear and non-linear systems, respectively [68]. The intersection of Bayesian filtering with machine learning has opened new frontiers. Bayesian filtering excels in dealing with uncertainty and noise in visual data, which is crucial in tasks like object recognition [69], object detection [70], gesture recognition [71], and scene understanding [72]. Recent breakthroughs by Yu et al.  [73], which introduced a recursive Bayesian filtering technique for remaining useful life estimation of degrading systems, and Su et al.  [74], which showcased improved performance in breast cancer prediction using a hybrid Bayesian network model. We hold the opinion that Bayesian filtering continues to play a pivotal role in advancing visual task processing. Its ability to manage uncertainty and integrate with emerging technologies positions it as a critical player in the future of image and video analysis.

Bayesian filtering deals with estimating a system’s hidden state based on a sequence of observations. In the context of TIR tracking, if we consider the target position as the hidden state and the raw infrared information from each frame as the observation, then TIR tracking essentially becomes a specialized application of Bayesian filtering. However, it should be noted that Bayesian filtering itself only offers a theoretical framework for state estimation and requires the establishment of precise system and observation models for practical application to a specific problem.

The system state and the observations are the most essential concepts in Bayesian filtering, and they have the following important characteristics:

1. 1.

   The system state is the target to be estimated and is not directly observable.

2. 2.

   The system state affects the observations that can be observed.

3. 3.

   The system state is Markovian with respect to the observations, i.e., the system state at the next moment is only related to the state at the current moment and is independent of both the system state and the observations at past moments.

A state space can describe the above system characteristics, which consists of the following two models:

The system model describes the relationship between the system state at adjacent moments $(t-1)$ and $t$, which is a Markov chain according to the Markovianity of the system state and can be expressed as:

where $s_{t}$ denotes the system state at moment $t$, and $z_{1:t-1}$ denotes the observations from moment 1 to $(t-1)$.

The observation model describes the effect of the system state on the observations. It is the conditional probability distribution $p(z_{t}|s_{t})$ of the observations $z_{t}$ when the system state is $s_{t}$.

Fig.1 illustrates the relationship between the system state, observations, system model, and observation model. It can be seen that the update of the system state is only related to the system state and the system model at the current moment, while the update of the observations is only associated with the system state and the observable model at the current moment. Over time, the system state changes and produces observations.

The objective of Bayesian filtering is to estimate the system state $s_{1:t}$, given a series of observations $z_{1:t}$, a known system model $p(s_{t}|s_{t-1})$, and an observation model $p(z_{t}|s_{t})$. This step of the estimation process is iterative, and each step requires an estimate of the current system state $s^{*}_{t}$. The objective function can therefore be formulated as follows:

where $s_{1}$ is the initial state.

The iterative formula between $p(s_{t-1}|z_{1:t-1})$ and $p(s_{t}|z_{1:t})$ can solve the objective function (Eq.2). The derivation is as follows:

Based on the total probability theorem, the following equation can be obtained as:

Combining Eqs.1 and 3, we get:

According to the Bayes rule, the following equation can be obtained as:

The iterative relationship can be obtained according to Eqs.4 and 5 as:

The denominator is independent of $s_{t}$ and can be regarded as a constant here. Therefore, the above equation can be simplified as:

Eq.7 is the iterative formula for $p(s_{t-1}|z_{1:t-1})$ to $p(s_{t}|z_{1:t})$. Here, the initial value $p(s_{1}|z_{1})$ is given, and it is only necessary to obtain $p(s_{t}|s_{t-1})$ and $p(z_{t}|s_{t})$ at each step to carry the estimation process forward, which are provided by the system and observation models, respectively. Therefore, solving the state estimation problem depends on the specific settings of the system model $p(s_{t}|s_{t-1})$ and the observation model $p(z_{t}|s_{t})$.

To solve the TIR tracking problem using Bayesian filtering, we define the system state $s_{t}$ at time $t$ as the motion data of the target object, i.e., the relative variation of the target center coordinates between $(t-1)$ and $t$. The reasons for this definition are as follows.

1. 1.

   The tracking method is concerned with the change in position due to the motion of the target object between consecutive TIR video frames, not the position itself. The next position can be obtained by adding the change to the current position.

2. 2.

   The size of the target object is different in TIR video sequences. Therefore, the same absolute change in position has different meanings for different target sizes. As a highly abstract variable, the system state should not depend on a specific target size, so it is more appropriate to use relative center variations.

In the TIR tracking problem, a rectangular box $G=(x,y;w,h)$ is generally used to annotate the target object in a two-dimensional image, where $(x,y)$ are the coordinates of the top-left corner of the rectangle, and $(w,h)$ is the width and height of the rectangular box, respectively. Therefore, the center coordinates of the rectangular box are calculated as:

The variation in the center coordinates of the rectangular box between $t-1$ and $t$ (where $t>1$) is as follows,

The relative change corresponding to the variation is as follows,

where $(\Delta cx_{t},\Delta cy_{t})$ is the system state $s_{t}$. During the tracking process, the position of the target object at time $T$ can be expressed as $\sum^{T}_{t=2}s_{t}^{*}$, where $s_{t}^{*}$ denotes the best estimate of the system state at time $t$.

For the observation $z_{t}$ in the Bayesian filtering framework, we define it as the raw infrared image of the $t$-th frame. The observation model estimates the distribution of the system state from the infrared information of the current frame. Thus, the tracking problem can be solved by estimating the optimal system state $\{s_{t}^{*}\,|\,t=1,2,\ldots,T\}$.

The system model is defined as the probability distribution $p(s_{t}|s_{t-1})$ of the current state $s_{t}$ under the condition that the system state $s_{t-1}$ at the previous frame. It is used to give an a priori estimate of the likelihood of the current state of the system based on the previous state, i.e., the prior distribution, before the observation $z_{t}$ acts at the current moment. We established the system model using 2D Brownian motion, which describes the tendency of a target object to move on its own without external influences. It is independent of the observation model and unrelated to infrared information. This situation is similar to the motion of particles in physics when they are not subject to external forces.

For a 2D Brownian motion $\mathcal{B}$, it satisfies the following properties:

1. 1.

   The initial value is 0, i.e., $\mathcal{B}_{0}=(0,0)$.

2. 2.

   The function $t\rightarrow\mathcal{B}_{t}$ is continuous at $t$ almost everywhere.

3. 3.

   The increment of $\mathcal{B}_{t}$ is independent, and for $t>s$, satisfies $(\mathcal{B}_{t}-\mathcal{B}_{s})$ independently of any $\mathcal{B}_{u}$, where $0\leq u\leq s$.

4. 4.

   The increment $(\mathcal{B}_{t}-\mathcal{B}_{s})$ obeys a 2D Gaussian distribution with mean $(0,0)$ and covariance $t\mathbf{I}$.

We model the DBF framework with 2D Brownian motion and define $\mathcal{B}_{t}$ as the target position $(x,y)$ at time $t$. The system state $s$ is the difference $(\Delta x^{\prime},\Delta y^{\prime})$ between two adjacent frames, which is also the increment of $\mathcal{B}_{t}$. Their relationship can be expressed as follows,

where the units of the difference $(\Delta x^{\prime},\Delta y^{\prime})$ are the scale of the DBF state space, not the pixel distance of the raw infrared image.

Since the initial target position $(x_{0},y_{0})$ is specified in the first video frame, we set this position as the origin to satisfy Property 1, i.e., $\mathcal{B}_{0}=(0,0)$. Although the computer calculates and stores the position $(x,y)$ in discrete form, we can assume that the position in DBF state space is continuous to satisfy Property 2. The fact that the pixel distances, in practice, are just discrete samples of the position in DBF space does not affect the effectiveness of our approach. In Property 3, $\mathcal{B}_{t}$ has independent increments implies that $(\mathcal{B}_{t_{1}}-\mathcal{B}_{s_{1}})$ and $(\mathcal{B}_{t_{2}}-\mathcal{B}_{s_{2}})$ are independent if $0\leq s_{1}<t_{1}\leq s_{2}<t_{2}$ — the combination of Eq.11 leads to the conclusion that $s_{t}$ and $s_{t-1}$ are independent. So the system model $p(s_{t}|s_{t-1})$ can be simplified as $p(s_{t})=p(\Delta x^{\prime},\Delta y^{\prime})$. According to Property 4, the increment $s_{t}$ obeys a 2D Gaussian distribution with mean $(0,0)$ and covariance $t\mathbf{I}$. Since the time interval $t$ is 1 here, the covariance can be expressed as $\mathbf{I}$. It further follows that both $\Delta x^{\prime}$ and $\Delta y^{\prime}$ obey a 1D Gaussian distribution $\mathcal{N}(0,1)$ and are independent of each other.

Based on the above analysis, the system model can be represented as follows,

It is worth noting that the scale units in DBF state space are not equivalent to the actual image scale. Therefore, we use the coefficients $\lambda_{x}$ and $\lambda_{y}$ to transform the unit scale from the image to the state space. The final system model is as follows,

where $\Delta x$ and $\Delta y$ is the target position difference in the raw image.

The observation model is used to evaluate the probability $p(z_{t}|s_{t})$ that the most recent observation $z_{t}$ occurs in the current system state $s_{t}$. It obtains a likelihood estimate of the system state after obtaining the observations. It is used to make a likelihood estimation of the system state after obtaining the observations, correcting the probability distribution predicted by the system model.

The observation model in this study is based on the detection methods. This is due to the many similarities between the two approaches as follows,

1. 1.

   Each candidate corresponds to each possible system state $s_{t}^{i}$.

2. 2.

   The current frame image is the observation $z_{t}$.

3. 3.

   The detection method generates a response score based on the feature similarity between the template and the candidate window in the current frame, and the more similar to the template, the higher the response score. The response scores are highly consistent with the role of the observation model output $p(z_{t}|s_{t})$.

Based on this analysis, we propose an observation model based on the tracking-by-detection method. As shown in Fig.2, the pipeline of the observation model mainly consists of two modules: feature extraction and classifier.

We choose VGG-M [75] as the network architecture for feature extraction. We use the activation value of the third convolutional layer as the deep feature for image representation, and all other parameters are kept consistent with the original VGG-M architecture, as shown in Table I. Inspired by MDNet [37], we use a neural network with three fully connected layers as the classifier. The tracking task involves only two categories, i.e., foreground (target object) and background, instead of the 1000 categories of the ImageNet classification task [44]. Moreover, the former uses low-level convolutional features, unlike the latter, which uses high-level convolutional features [76]. Therefore, we reduce the number of units in the intermediate hidden layer from 4,096 to 512 dimensions for the general classification task. This also reduces the costs of computation and storage. The specific network parameters are shown in Table II, where the fully connected layer is actually implemented with a $1\times 1$ convolutional layer, and the last layer is a softmax function that outputs a 2D vector $\mathbf{v}=(v_{1},v_{2})$, where $v_{1}$ and $v_{2}$ denote the probability that the candidate is foreground and background, respectively. Since $v_{1}+v_{2}=1$, we take $v_{1}$ as the response score.

The final output of the observation model is the likelihood probability $p(z_{t}|s^{i}_{t})$, so it is also necessary to transform the response score into a probabilistic form. We use a weighted softmax function to perform the following transformation as follows,

where $v_{i}$ is the response score of the system state $s^{i}_{t}$. It can be seen that the higher the response score is, the higher the likelihood probability is. $\alpha^{i}$ is a penalty hyperparameter that obeys the Gaussian distribution and controls the impact of the response score on the probability distribution, i.e., the closer $s_{t}^{i}$ is to the center of the previous frame, the larger $\alpha^{i}$ is.

Our TIR object tracking method, DBF, was developed using the MatConvNet [77] toolbox and trained on both RGB datasets such as COCO [45], LaSOT [49], and GOT-10k [48], and the TIR dataset LSOTB-TIR [14]. All the experiments were conducted on a cloud server with an Intel^® Xeon^® E5-2680 v4 CPU @ 2.4GHz CPU with 128GB RAM, and a NVIDIA^® GeForce RTX^TM 2080 Ti GPU with 11GB VRAM. We trained the observation model using Stochastic Gradient Descent (SGD) with a batch size of 8 and a momentum of 0.9. As described in Section III-D, we used a modified VGG-M [75] as the base feature extractor. The classifier’s settings adhered to the default configurations of MDNet [37]. The initial training phase of the observation model spanned 60 epochs on the RGB datasets, with the learning rate exponentially decayed from $10^{-2}$ to $10^{-5}$. This was followed by a fine-tuning phase of 30 epochs on the TIR dataset, setting the learning rate at $10^{-3}$ and the weight decay at $10^{-5}$.

Datasets: PTB-TIR [15] dataset comprises 60 manually annotated infrared pedestrian sequences, amassing more than 30,000 frames in total. Each sequence has nine attribute labels for the attribute-based evaluation. LSOTB-TIR [14] dataset is notably larger, offering 1400 infrared video sequences with high-quality annotations, totaling in excess of 600,000 frames. It features five varieties of moving objects – people, animals, vehicles, aircraft, and boats – in four distinct scenarios and is evaluated on 12 challenging attributes, thereby making it the most extensive TIR dataset to date.

Evaluation metrics: Three distinct metrics are employed in LSOTB-TIR [14] dataset to evaluate the performance of TIR trackers. Precision is defined as the ratio of frames in which the Center Location Error (CLE) falls below a specified threshold (20 pixels), relative to the total number of frames. Normalized Precision adjusts for the resolution and size of the predicted bounding box, mitigating the impact of varying target sizes. The effectiveness of TIR trackers is assessed using the Area Under Curve (AUC) of Normalized Precision within the range of 0 to 0.5. Success is measured by the proportion of frames where the overlap ratio (OR) between the predicted and ground truth bounding boxes exceeds a predefined threshold. For the PTB-TIR [15] dataset, precision and success are used to assess the performance of the TIR trackers.

In this section, we first analyze the proposed DBF method on two benchmark datasets to evaluate the effectiveness of the feature extractor network architecture and training dataset size.

Network architecture: Among existing TIR object tracking methods, the commonly used CNN architectures are AlexNet [78], VGG-M [75], and VGG-16 [75]. Table III presents the results of the proposed DBF using these different architectures as the feature extractor. Notably, VGG-M surpasses AlexNet and VGG-16, registering the best scores across all evaluation metrics on both benchmark datasets. AlexNet, an earlier architecture, is smaller in size and offers faster computation but lacks in performance. It is typically employed in early tracking methods where speed is prioritized. VGG-M, in comparison, is wider than AlexNet. The initial two convolutional layers of AlexNet use $5\times 5$ and $3\times 3$ kernels, whereas VGG-M uses $7\times 7$ and $5\times 5$ kernels in its first two layers. Additionally, VGG-M boasts more convolutional channels in layers 2, 3, and 4 than AlexNet and incorporates batch normalization to enhance training efficiency. In practice, many TIR tracking methods utilize only the shallow features of VGG-M, which allows for rapid computation while delivering robust performance, making it a popular choice among convolutional feature extractors. The characteristic of VGG-16 is that it is relatively deep, and replaces a single layer of large convolution kernels with multiple layers of $3\times 3$ small convolution kernels. VGG-16, known for its depth and use of smaller $3\times 3$ convolution kernels, does not perform as well in tracking tasks as it does in classification. This could be due to tracking tasks relying more on shallow features.

Training dataset size: To train the observation model of DBF, a mix of RBG and TIR datasets was used. The RGB datasets encompassed COCO [45], LaSOT [49], and GOT-10k [48]. COCO is a comprehensive dataset designed for image detection and semantic segmentation, containing over 330,000 images and 1.5 million objects. LaSOT, a newer benchmark for long-term visual tracking, includes 1,400 video sequences, each averaging 2,500 frames, spanning 70 class categories with 16 sequences per category in its training set. GOT-10k, another challenging benchmark, consists of 10,000 video sequences for training purposes. The training subset of the LSOTB-TIR [14] was also utilized. Table IV illustrates the effectiveness of DBF using different combinations of training datasets. The combined use of RGB and TIR datasets achieves better results than training solely with either RGB or TIR datasets. This indicates that the integration of RGB and TIR datasets exploits the unique properties of RGB and TIR images, thereby enhancing the robustness of feature extraction for TIR object tracking.

To evaluate our proposed DBF, we compared it with state-of-the-art TIR tracking methods, including CFNet [79], DSiam [80], ECO-deep [20], ECO-HC [20], ECO-STIR [81], HSSNet [82], MDNet [37], MLSSNet [22], MMNet [24], SiamFC [17], SiamTri [18], SRDCF [83], TADT [84], VITAL [85], and UDT [86].