Learning to Calibrate for Reliable Visual Fire Detection

Calibration can be classified into post-calibration and online calibration. Post-calibration refers to the process of re-mapping the predictions of a pre-trained model to yield more accurate probabilities. Common post-calibration techniques include Temperature Scaling, Vector Scaling, and others [17]. In contrast, online calibration involves constraining predictive uncertainty during the model’s training process, allowing the model to generate credible predictions directly.

We focus on multi-class tasks in visual fire detection, where the model takes a single image $X$ as input, and the corresponding label is denoted as $Y\in\{1,\ldots,K\}$, with $K$ representing the number of classes. The model processes the input images and, after passing through the final softmax activation function, outputs the predicted probabilities for each class $\{\hat{p}_{1},\hat{p}_{2},\ldots,\hat{p}_{K}\}$. The largest predicted probability is denoted as $\hat{P}$, and the class corresponding to $\hat{P}$ is the model’s predicted class, denoted as $\hat{Y}$. Therefore, $\hat{P}$ represents the model’s confidence for the sample belonging to class $\hat{Y}$.

The ideal calibration result for a multi-class fire detection model is given by

The equation implies that for all samples, the model’s confidence in its predictions should be numerically consistent with its true classification accuracy. Specifically, if $m$ samples all have a confidence level $p$, then the model should correctly classify $m\times p$ of these samples. The discrepancy between the model’s predicted confidence and the actual observed accuracy across different confidence levels is what the calibration error measures. However, since $\hat{P}$ in equation 1 is a continuous random variable, it is challenging to verify the equation’s validity based on a finite number of samples. Therefore, binning or other approximation methods are typically employed to address this issue.

Uncertainty is an abstract concept which must be assessed through either visual or quantitative methods. A reliability diagram is a visual method for reflecting the uncertainty of a model by representing the true classification accuracy as the function of confidence. The implementation steps are as follows: First, test samples are input into the trained model to obtain corresponding predicted probabilities and classification predictions. Next, the interval $[0,1]$ is divided into $M$ sub-intervals, and the predicted probabilities $\hat{P}$ are assigned to one of the $M$ intervals. Let $B_{m}$ represents the set of samples whose predicted probabilities fall within the interval $I_{m}=(\dfrac{m-1}{M},\dfrac{m}{M})$. The predicted accuracy for all samples in the m-th interval can be expressed as

where $\hat{y_{i}}$ represents the predicted category of sample $i$, $y_{i}$ represents the true category of sample $i$, $B_{m}$ represents the set of samples falling in the $m$-th interval, and $acc(B_{m})$ can be seen as an unbiased estimate of $P(\hat{Y}=Y|\hat{P}\in I_{m})$. The average confidence of all samples in the $m$-th sub-interval can be expressed as

where $\hat{p}_{i}$ represents the predicted probability, also known as the confidence, of sample $i$, and $conf(B_{m})$ can be considered as an approximation of the value of $p$ on the right side of equation 1.

Therefore, equation 1 can be approximated as $acc(B_{m})=conf(B_{m})$, meaning that, in the case of ideal calibration, the reliability diagram should display as the identity function. Taking $M=10$, the reliability diagram corresponding to the ideal calibration is shown in Figure 2. The closer the curve (in red) is to the diagonal line (in black), the better the calibration performance. In the diagram, the height of the pink bars represents the average confidence of samples in each sub-interval, while the height of the purple bars reflects the classification accuracy of samples in the corresponding sub-interval. Ideally, the two areas coincide completely.

Reliability diagrams provide an intuitive means of reflecting a model’s uncertainty. However, when the differences between two reliability diagrams are subtle, as illustrated in Figure 3, it is hard to assess which model can provide more reliable decisions.

In such cases, representing the model’s uncertainty using a scalar value proves to be more practical. A widely adopted method for quantification is illustrated as

This method of uncertainty quantification is derived from equation 1, which assesses the predictive uncertainty by evaluating the gap between the model’s confidence in its predictions and the true classification accuracy. This gap is typically approximated using the ECE[20] metric. The calculation of ECE follows a process similar to the construction of reliability diagrams: the interval $[0,1]$ is divided into $M$ sub-intervals, and within each sub-interval, the weighted average of the differences between the average accuracy and confidence is computed, as outlined in equation 5, where $n$ denotes the total number of samples. ECE evaluates the discrepancy between the predicted confidence and the true classification accuracy. A smaller ECE indicates better calibration performance of the model.

We consider the approach of online calibration in visual fire detection, which involves constraining the model’s credibility during training. One possible method is to use ECE as a loss function. However, the calculation of the accuracy $\textit{acc}(B_{m})$ in the ECE metric involves the 0-1 indicator function, as described in equation 2. Therefore, ECE is non-differentiable, making it unsuitable for direct use as a loss function during optimization.

The sigmoid function, with a range of (0, 1), can map any real number to this interval and exhibits a monotonically increasing behavior, facilitating a smooth transition between 0 and 1. Consequently, we propose approximating the indicator function with the sigmoid function. After this adjustment, the accuracy calculation is modified as presented in equation 6. This converts the ECE metric into a differentiable ECE Loss, without altering the underlying calculation logic.

where the sigmoid function S(x) is given in equation 7, and its corresponding curve is depicted in Figure 4.

When $\hat{p}_{i}$ varies from 0 to 1, the curve of $\textit{acc}(B_{m})$ exhibits a smooth and differentiable profile, as depicted in Figure 5.

In this study, we combine ECE Loss with cross-entropy loss (NLL Loss) to jointly supervise the training of a multi-class fire detection model. Initially, we observe the relative magnitudes of NLL Loss and ECE Loss during training. Based on their proportions, we set the expected weight $\gamma_{E}$ for ECE Loss, while fixing the weight coefficient of NLL Loss at 1.0. At the early stage of training, the model’s classification performance is not stable, and thus the constraining effect of ECE Loss on model uncertainty is relatively weak. To balance the model’s classification accuracy and reliable decision, we draw on the curriculum learning [21]. Specifically, we gradually increase the weight of ECE Loss as the number of training epochs progresses, until it reaches the predefined value of $\gamma_{E}$. This strategy ensures that as the model becomes more confident and its accuracy improves, ECE Loss progressively contributes more to reducing uncertainty.

For DFAN model, the ratio of NLL Loss to ECE Loss is approximately 1:20, which leads to the choice of $\gamma_{E}=0.05$; for EdgeFireSmoke model, the ratio of NLL Loss to ECE Loss is approximately 5:1, so $\gamma_{E}=5$. The overall loss function used during model training can be expressed by

where $L_{n}$ represents the NLL Loss, and $L_{e}$ the ECE Loss. When calculating ECE Loss, the number of sub-intervals $M$ is set to 10. The variable $c_{e}$ denotes the current training epoch, $s_{e}$ denotes the epoch at which ECE Loss is first incorporated into the loss function, and $N$ is the total number of epochs the model is trained for. Based on observations from the experimental process, as the training epoch increases, the model’s classification accuracy gradually stabilizes and the weight of ECE Loss gradually increases, which ultimately approaches the expected weight. As a consequence, the constraining effect of ECE Loss on the model’s uncertainty becomes more pronounced, encouraging the model to refine its uncertainty estimation and improve its overall calibration.

The DFAN dataset [22] is sourced from videos on platforms such as YouTube, Facebook, and disaster emergency management agencies, comprising a total of 3,803 images spread across twelve categories. The dataset is split into training, validation, and testing sets in a ratio of 7:2:1. The distribution of images across the different categories is presented in Table 1, and some example images from each category are shown in Figure 6.

The EdgeFireSmoke dataset [23] consists of wildfire images captured by drones and is organized into four categories: Burned-area images, which typically feature blackened ground or withered tree trunks; Fire-smoke images, where the smoke is depicted as black or white; Fog-area images, characterized by blurred visuals and difficulty in distinguishing objects within the environment; and Green-area images, representing normal environments without the presence of the conditions mentioned above. The dataset contains a total of 49,452 images, with the data split into training, validation, and testing sets in a 2:3:5 ratio. The distribution of images across the different categories is shown in Table 2, and some example images from each category are illustrated in Figure 7.

We employ five evaluation metrics to assess the performance of the multi-class fire detection models: precision, recall, F1 score, accuracy, and the ECE metric. The first four metrics are designed to evaluate the model’s classification performance, while the ECE metric is intended to measure the decisions’ reliability.

In the multi-classification task of this paper, precision, recall, F1 score, and accuracy are first calculated for each individual class, and then the average values across all classes are taken as the overall metrics. Besides, to measure the model’s predictive uncertainty, the ECE metric is employed. The calculation of accuracy follows the method outlined in equation 2. When plotting the reliability diagram and calculating ECE on the test data, the number of bins $M$ is set to 15 to provide a detailed evaluation of the model’s calibration across different confidence levels.

DFAN and EdgeFireSmoke models are implemented using the TensorFlow framework, and the training procedures largely adhere to the parameter settings outlined in the original papers. For the DFAN model, the training process is conducted over 50 epochs. To ensure the effective calibration of ECE Loss, and considering the relatively small size of the DFAN dataset, the batch size is increased to 32. Additionally, the input images are resized to 299×299 pixels. The model is optimized using the SGD optimizer with a learning rate set to 0.001. For the EdgeFireSmoke model, the training is performed over 30 epochs. To fully exploit the calibration effect of ECE Loss, the batch size is set to 128, and the images are resized to 224×224 pixels. The Adam optimizer is used with a learning rate of 0.001.

The model trained with NLL Loss is referred to as the vanilla model, while the model trained by combining NLL Loss and ECE Loss is referred to as the calibrated model. The vanilla DFAN model is denoted as $\textit{DFAN}_{\textit{nll}}$, and the calibrated DFAN model is denoted as $\textit{DFAN}_{\textit{cali}}$. Experiments results have demonstrated that the best calibration effect is achieved when $s_{e}=0$, meaning that ECE Loss is incorporated from the start of training. The experimental comparison between the vanilla and calibrated DFAN models is shown in Table 3. Compared to the $\textit{DFAN}_{\textit{nll}}$ model, the $\textit{DFAN}_{\textit{cali}}$ model exhibits a significant reduction in uncertainty, with only a minor decrease of less than 0.7% in classification accuracy. The reliability diagrams of the vanilla and calibrated models are shown in Figure 8. The left diagram represents the reliability diagram for model $\textit{DFAN}_{\textit{nll}}$. In the confidence intervals [0.2, 0.4] and [0.6, 0.8], a noticeable gap exists between the model’s predicted confidence and its actual prediction accuracy. The function curve (the red curve in the diagram) deviates significantly from the perfect prediction (represented by the black dashed line), indicating a higher level of model uncertainty. In contrast, the right diagram shows the reliability diagram for model $\textit{DFAN}_{\textit{cali}}$. In the interval [0.6, 0.8], the model’s confidence and prediction accuracy are more closely aligned, with the function curve closely following the diagonal line. In the interval [0.2, 0.4], the prediction results have also improved, indicating better calibration. The calibrated model provides more reliable predictions, with better alignment between predicted confidence and classification accuracy.

The experimental comparison between the vanilla and calibrated EdgeFireSmoke models is shown in Table 4. The vanilla EdgeFireSmoke model is denoted as $\textit{Edge}_{\textit{nll}}$. In the online calibration experiments for this model, two experimental schemes demonstrate effective calibration effects. The first experimental scheme sets $s_{e}=0$, and the resulting model is denoted as $\textit{Edge}_{\textit{cali1}}$, corresponding to the second row of experimental results in Table 4; the second scheme sets $s_{e}=10$, and the resulting model is denoted as $\textit{Edge}_{\textit{cali2}}$, corresponding to the third row of experimental results in the table. Although the vanilla model already demonstrated low uncertainty, the introduction of ECE Loss still results in a decrease in model uncertainty, and the loss in model accuracy is controlled within 0.5%. This indicates that the inclusion of ECE Loss successfully mitigates uncertainty without significantly sacrificing classification accuracy. The reliability diagrams of both the vanilla and calibrated EdgeFireSmoke models are shown in Figure 9. In the interval [0.6, 1.0], the predicted accuracy and average confidence in both $\textit{Edge}_{\textit{cali1}}$ and $\textit{Edge}_{\textit{cali2}}$ models are much closer, highlighting the effectiveness of ECE Loss in reducing the model’s tendency towards overconfidence. These results demonstrate that ECE Loss can improve the decisions’ reliability by ensuring that the model’s confidence aligns more accurately with its prediction accuracy.

Additionally, to balance the model’s classification accuracy and reliable decision, a dynamic loss function design is proposed, where the weights of the loss functions change over time. This approach is inspired by the concept of curriculum learning, which allows the model to progressively incorporate more complex tasks as it stabilizes on simpler tasks. To verify the effectiveness of this approach, we consider incorporating ECE Loss from the beginning of the training process, with both loss functions’ weights fixed at their expected values throughout training. This can be specifically represented by

When training the model, the weight coefficient $\gamma_{E}$ for ECE Loss remains constant, ensuring that the magnitudes of both loss functions are kept consistent and exert an equal constraining effect on the model’s training process. Under this setting, the training results of the DFAN model are denoted as $\textit{DFAN}_{\textit{w/o cl}}$, and those of the EdgeFireSmoke model are denoted as $\textit{Edge}_{\textit{w/o cl}}$. The comparative experimental results are shown in Table 5 and Figure 10. The results demonstrate that, whether in terms of model classification accuracy or decision reliability, the models trained using curriculum learning approach outperforms the models with a constant weight setting for ECE Loss. This suggests that during the early stage of training, when the model’s classification accuracy is still developing, introducing ECE Loss could even hinder the model’s ability to learn effective classification patterns. Following the curriculum learning, the model first focus on the classification accuracy and later on the decision reliability. This gradual transition leads to more reliable and accurate predictions.