Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

The ToDayGAN [24] model introduces three discriminators to improve the translation performance of visible images from nighttime to daytime. ToDayGAN-TIR [9] adapts ToDayGAN to improve the performance of NTIR image colorization. To avoid dot artifacts, ToDayGAN-TIR replaces the last two instance normalization layers of the decoder with group normalization [35] layers and combines the total variance [36] loss $\mathcal{L}_{tv}$ to reduce the noise of the translation results. To improve the representation ability of cycle-consistency loss $\mathcal{L}_{cyc}$ for NTIR images, it introduces SSIM [37] loss based on the original L1-norm loss. In addition, it introduces the spectral normalization [38] layer into the discriminator to make the model training more stable. Similar to ToDayGAN, it chooses relativistic loss [39], adapted to least-squares GAN loss, as adversarial loss $\mathcal{L}_{adv}$.

To predict the semantic mask of NTIR images without available annotation, which is a necessary component for semantic consistency loss, we need high-quality fake NTIR images and corresponding DC image pseudo-labels to train the segmentation network for NTIR domain. Therefore, we first introduce SGA [9] loss $\mathcal{L}_{sga}^{ori}$ to reduce the edge distortion during translation based on the ToDayGAN-TIR model. SGA loss encourages the ratio of the gradient of the synthesized image at the edge of the original image to the maximum gradient to be greater than a given threshold. In addition, to further obtain high-quality fake NTIR images, we introduce two regularization terms in the SGA loss of domain A, the monochromatic regularization term and the temperature regularization term. As the output fake NTIR image is three-channel data, the monochromaticity regularization term aims to encourage the values of the three channels to be the same. For the unnatural situation where the mean value of the pedestrian area in the fake NTIR image is extremely small, the temperature regularization term is responsible for encouraging the minimum value of the pedestrian area to be no less than the mean value of the road area. This regularization term is inspired by the observation that the body temperature of pedestrians is usually higher than the mean temperature of the road area during nighttime conditions.

Concretely, similar to [40], we first define the channel maximum operation as a mapping $cmax:\mathbb{R}^{C\times H\times W}\to\mathbb{R}^{H\times W}$. Then, given the input $I\in\mathbb{R}^{C\times H\times W}$, the output $O\in\mathbb{R}^{H\times W}$ of the channel maximum operation at position $\left(i,j\right)$ can be expressed as

Next, the monochromatic regularization term can be denoted as

For the temperature regularization term, given the mean value of the road region of the fake NTIR image, denoted as $\bar{v}_{road}^{fb}$, and the minimum value of the pedestrian region of the fake NTIR image, denoted as $\tilde{v}_{ped}^{fb}$, the temperature regularization term can be represented as

where the denominator is intended to normalize the output in the range $\left[0,1\right)$, and $\varepsilon$ is a small value to avoid dividing by zero. Ultimately, the improved SGA loss can be expressed as

Combining the above two adjustments, we obtain the variant model ToDayGAN-NTIR, which serves as the baseline for MornGAN.

To obtain segmentation pseudo-labels for both domains, we first introduce a novel semantic denoising process that utilizes the class-specific low-rank properties of the original image to remove noisy labels that deviate from the distribution. For the specific category $y_{1}$ in the dataset, given the coarse pseudo-labels $M_{all}\in\mathbb{R}^{H\times W}$, the input image $I_{x}\in\mathbb{R}^{C\times H\times W}$, and the set $Z_{y_{1}}=\left\{y_{12},y_{13},\cdots,y_{1m}\right\}$ of confusion categories of $y_{1}$, we can obtain a more trustworthy binary mask $\hat{M}_{y_{1}}\in\mathbb{R}^{H\times W}$ through the semantic denoising process. Specifically, for a given category $y_{k}$, we first compute its binary mask $M_{y_{k}}\in\mathbb{R}^{H\times W}$ and average pixel feature $f_{y_{k}}\in\mathbb{R}^{C\times 1}$. Subsequently, we define the matrix of feature distances from pixels belonging to category $y_{1}$ to category $y_{k}$ as $Q_{y_{1}y_{k}}$, and the value of $Q_{y_{1}y_{k}}$ at position $u$ can be formulated as

where $\odot$ denotes element-wise multiplication with channel-wise broadcasting, and the first term of the multiplication aims to ignore the feature distances of the locations that do not belong to category $y_{1}$, while the second term is used to calculate the Euclidean distance between features. Ultimately, the value of $\hat{M}_{y_{1}}$ at position $u$ can be given by

where $\mathbb{I}\left\{\cdot\right\}$ is the indicator function (i.e., output 1 when the condition is met, and otherwise 0), and the equation in the indicator function determines whether the label at that location is noise by comparing the magnitude of the intra-class distance with that of the inter-class distance. In sum, the above semantic denoising process can be expressed using the mapping $\mathcal{SDP}\left(\cdot\right)$ as

If there are multiple categories to be denoised, each category is updated with its own average feature after denoising, which enables a more reasonable estimation of the inter-class distance afterward. Two examples of SDP are shown in Fig. 4, and the fifth column shows the result of SDP on the fourth column.

As there is no segmentation label for DC images and the domain-adapted semantic segmentation methods still suffer from high-confidence noise, we directly utilize the existing segmentation models trained on different datasets to jointly predict the pseudo-labels of DC images. Concretely, we choose Detectron2 [33], a panoptic segmentation model trained on the MS COCO dataset [41], and HMSANet [34], a semantic segmentation model trained on the Cityscape dataset [42]. Due to the difference in training data, the Detectron2 model is good at object contour segmentation but is weaker than the HMSANet model for segmentation of traffic scene categories, as shown in the first row in Fig. 4. Therefore, we take the intersection of the predictions of HMSANet and Detectron2 as the pseudo-labels of the pedestrian, car, and building categories, and the predictions of HMSANet for the other categories. The pseudo-labels obtained by fusion are noted as $M_{F}^{A}$.

However, we observe that the segmentation results after upsampling usually show shifting of edges and inflation of semantic regions due to the pooling and strided convolution operations in deep neural networks. In addition, there are significant low-rank properties (e.g., color homogeneity) within some background classes (i.e., tree, sky, and pole) in the visible domain with large inter-class differences. Accordingly, we exploit the distributional properties of these categories to reduce the noise in the pseudo-labels. Specifically, assuming that the categories of tree, pole, and sky are denoted as $t$, $o$ and $s$, respectively, the pseudo-labels of these three categories are refined sequentially using the semantic denoising process mentioned above; that is, $\mathcal{SDP}\left(M_{F}^{A},x_{a},Z_{t}\right)$, $\mathcal{SDP}\left(M_{F}^{A},x_{a},Z_{o}\right)$ and $\mathcal{SDP}\left(M_{F}^{A},x_{a},Z_{s}\right)$. Because the semantic edges of trees and poles usually expand into the sky region in the predicted segmentation masks, the set of confusion categories $Z_{t}$, $Z_{o}$, and $Z_{s}$ are $\left\{s\right\}$, $\left\{s\right\}$, and $\left\{t,o\right\}$, respectively. After denoising, we can obtain the pseudo-labels $M_{PL}^{A}$ of the DC image. An example image after denoising is shown in the fifth column of the first row in Fig. 4, and the final pseudo-labels can significantly reduce the predicted noise.

With the obtained pseudo-labels of DC images, we can train the segmentation network $S_{A}$ with input $x_{a}$, and $S_{B}$ with input $G_{AB}\left(x_{a}\right)$. After that, we devise an online semantic distillation module to jointly predict the pseudo-labels of NTIR images by using $S_{A}$ and $S_{B}$. The online semantic distillation module consists of a label mining process and a semantic denoising process, where the former mines the high-confidence part of the network prediction intersection as the coarse pseudo-labels, and the latter denoises the pseudo-labels based on the category distribution properties. Concretely, we first define the probability tensor of outputs $S_{B}\left(x_{b}\right)$ and $S_{A}\left(G_{BA}\left(x_{b}\right)\right)$ as $V^{rb}\in\mathbb{R}^{N_{c}\times H\times W}$ and $V^{fa}\in\mathbb{R}^{N_{c}\times H\times W}$, respectively, whose values at the $c_{th}$ channel and at position $u$ denote the probability that the position belongs to category $c$, denoted as $V_{c,u}^{rb}$ and $V_{c,u}^{fa}$. Moreover, $N_{c}$ denotes the number of categories. Considering the difference in the number of categories and the fact that GAN models usually translate object regions in IR images into background regions to enhance the realism of synthesized images [9], label mining using the same threshold for all categories is suboptimal. Therefore, we design two thresholds, $\theta_{fg}$ and $\theta_{bg}$, to extract the pseudo-labels of the foreground category $\mathcal{C}_{fg}$ and background category $\mathcal{C}_{bg}$, respectively. Then, the category of pseudo-labels at location $u$ obtained by the label mining process, denoted as $\left(M_{LM}^{B}\right)_{u}$, can be represented as

And $\eta\left(\cdot\right)$ is a category-dependent piecewise function:

Due to the presence of high confidence noise in the label mining results, we exploit the semantic denoising process to suppress the noisy labels similar to how we handle pseudo-labels of DC images. Considering that the body temperature of pedestrians is usually significantly higher than some of the background categories (e.g., trees) in the nighttime environment, we add denoising for the pedestrian region to the three background categories (i.e., trees, poles, and sky) mentioned in the previous subsection. Specifically, we first abbreviate the pedestrian category as $p$. Then, we use Eq. (7) to denoise the four categories of sky, pole, pedestrian and tree in turn, i.e., $\mathcal{SDP}\left(M_{LM}^{B},x_{b},Z_{s}\right)$, $\mathcal{SDP}\left(M_{LM}^{B},x_{b},Z_{o}\right)$, $\mathcal{SDP}\left(M_{LM}^{B},x_{b},Z_{p}\right)$, $\mathcal{SDP}\left(M_{LM}^{B},x_{b},Z_{t}\right)$. Considering the spatial distribution of the categories, the set of confusion categories $Z_{s}$, $Z_{o}$, $Z_{p}$, and $Z_{t}$ is set to $\left\{t,o\right\}$, $\left\{s\right\}$, $\left\{t\right\}$, and $\left\{s,p\right\}$, respectively. After denoising, we can obtain the pseudo-labels of the NTIR image, denoted as $M_{PL}^{B}$. An example diagram of the online semantic distillation module is shown in the second row in Fig. 4.

Pseudo-labels of both domains are used not only for supervising the training of the segmentation network, but also for encouraging semantically invariant image translation. Both stages require segmentation loss as the optimization objective. Due to the uneven distribution among categories, we utilize a modified pixel-level cross-entropy loss [43] as the segmentation loss for the three branches (i.e., $\mathcal{L}_{seg}^{fb}$, $\mathcal{L}_{seg}^{rb}$, and $\mathcal{L}_{seg}^{fa}$), which assigns larger weights for smaller sample categories. Thanks to the sharp edges in the DC images and the relatively complete semantic edges in the pseudo-labels, we can improve the edge segmentation performance of model $S_{A}$ by introducing boundary loss. Referring to [44], the absolute value of the difference between the original image and the result after average pooling is taken as the spatial gradient. Then, we can calculate the difference in the spatial gradient between the predicted semantic probability tensor and the label, whose absolute value is taken as the boundary loss. Thus, the loss $\mathcal{L}_{seg}^{ra}$ consists of a modified pixel-level cross-entropy loss [43] and a boundary loss [44], where the weight of the boundary loss is empirically set to 0.5. Ultimately, the complete segmentation loss can be expressed as

where $\lambda_{ra}$, $\lambda_{fb}$, $\lambda_{rb}$, and $\lambda_{fa}$ are binary (i.e., either 0 or 1) loss weights used to switch learning stages.

The memory-guided sample selection strategy aims to select cross-domain sample pairs containing objects of the same categories for collaborative learning. Due to the lack of semantic labels of NTIR images, online memorization of semantic masks of NTIR images is necessary. Therefore, as shown in Fig. 3, after the weights of the segmentation network are fixed, we leverage the online semantic distillation module to infer the pseudo-label $M_{PL}^{B}$ of the NTIR image, which is subsequently stored in the memory unit. After the semantic masks of all NTIR images are stored, the sample selection strategy works to select the appropriate NTIR images for a given DC image based on the distribution similarity.

As the goal is to improve the colorization performance of small sample categories, the focus is on the similarity of the distribution of cross-domain sample pairs in small sample categories. Concretely, given the semantic mask $M_{PL}^{A}$ of a DC image, we first calculate the percentage of regions corresponding to each small sample category relative to the full image, and then subtract the mean value to obtain the vector $f_{d}^{A}$ as the semantic distribution feature. Similarly, we can obtain the semantic distribution feature $f_{d}^{B}$ of any NTIR image. Further, the similarity of the distribution between $f_{d}^{A}$ and $f_{d}^{B}$, denoted as $d_{AB}$, can be expressed using the folldue cosine similarity:

where $\left\|\cdot\right\|_{2}$ represents the L2 norm.

Although we can use similarity to find the NTIR image with the most similar (i.e., top-1 selection) distribution for the given DC image, the incompleteness of $M_{PL}^{B}$ may lead to frequent selection of a particular NTIR image, which can cause overfitting of the model. To avoid this problem, the top-1 selection strategy is relaxed to random sampling from the top-k candidates, which means that one of the $k$ most similar NTIR images is randomly selected for collaborative learning with the given DC image. Unlike the popular learning method of random sample pairs in GAN models [2], the proposed collaborative learning approach can pave the way for subsequent category-aware cross-domain constraints.

With the selection of cross-domain sample pairs containing small sample categories, an ACA loss is designed to further encourage the similarity of feature distributions within classes. Due to the complexity of the constituent parts of object categories, characterizing the texture of objects with a single mean feature is sub-optimal. Accordingly, we first use Kmeans [45, 46] to extract the features of the components of the small sample categories of the real DC image. Then, the inner product between each component feature and the original feature is used to characterize the response map or co-attentive map of that component feature. Ultimately, the distance between the corresponding response maps of the fake and DC images is used to measure the similarity of the feature distributions. Specifically, we first denote the features of the real DC image and the fake DC image after the encoder as $F^{ra}\in\mathbb{R}^{C\times h\times w}$ and $F^{fa}\in\mathbb{R}^{C\times h\times w}$, respectively, and their corresponding segmentation pseudo-labels as $M^{A}\in\mathbb{R}^{h\times w}$ and $M^{B}\in\mathbb{R}^{h\times w}$, respectively. Given any category $c_{k}$ in the small sample category set $\mathcal{C}_{ss}$, we can obtain the binary masks $M_{c_{k}}^{A}$ and $M_{c_{k}}^{B}$ of this category in two domains. Then, let the sum of non-zero elements in $M_{c_{k}}^{A}$ be $N_{c_{k}}^{A}$, the feature matrix $F_{c_{k}}^{ra}\in\mathbb{R}^{C\times N_{c_{k}}^{A}}$ corresponding to the category $c_{k}$ in the real DC image can be denoted as

where $\mathcal{E}\left(\cdot\right)$ denotes the operation of first transforming the input into a matrix of $C$ rows $h\times w$ columns and then extracting the columns in which the sum of absolute values is non-zero. Subsequently, we utilize Kmeans to obtain the clustering feature matrix $U_{c_{k}}^{ra}\in\mathbb{R}^{N_{u}\times C}$, where each row denotes the features of the cluster centroids, and $N_{u}$ denotes the number of clusters. Afterward, we can obtain the response map $Y_{c_{k}}^{ra}\in\mathbb{R}^{N_{u}\times N_{c_{k}}^{A}}$ of all centroid features, which can be expressed using the cosine similarity between features as follows:

Then, we reshape $Y_{c_{k}}^{ra}$ into $\tilde{Y}_{c_{k}}^{ra}\in\mathbb{R}^{N_{u}\times N_{c_{k}}^{A}\times 1}$ and $\hat{Y}_{c_{k}}^{ra}\in\mathbb{R}^{N_{c_{k}}^{A}\times N_{u}\times 1}$. Further, the mean value of the maximum response of each location for all centroid features can be formulated as

Correspondingly, the mean value of the maximum response of each cluster for the features at all locations can be expressed as

Similarly, we can compute $\mu_{c_{k}}^{fa}$ and $\tau_{c_{k}}^{fa}$ using $F^{fa}$ and $M_{c_{k}}^{B}$. Thus, the ACA loss for category $c_{k}$ can be presented as

where $\varphi_{l}$ and $\varphi_{g}$ are the thresholds for controlling the local (i.e., individual centroid features) and global (i.e., distribution of centroid features) similarity between the features of the synthesized image and the centroid features, respectively. At last, the ACA loss for all small sample categories can be shown as

where $N_{ssc}$ denotes the total number of small sample categories.

The FLIR and KAIST datasets are two commonly used benchmarks for the NTIR2DC task. The FLIR Thermal Starter Dataset [13] provides TIR images with bounding box annotations for training and validation of the object detection model, while the reference RGB images are unannotated. Through the same data split as in [9], we finally obtain 5447 DC images and 2899 NTIR images for training, while an additional 490 NTIR images are used for testing.

The KAIST Multispectral Pedestrian Detection Benchmark [14] provides coarse-aligned RGB and TIR image pairs, which contain both daytime and nighttime conditions. Folldue [9], the training set contains 1674 enhanced DC images and 1359 NTIR images, and an additional 500 NTIR images are used as the test set to evaluate the semantic and edge consistency. For the pedestrian detection experiments, the sample size of the test set is 611.

In order to remove the black areas on both sides in some images, according to [9], we first resize the training images to a resolution of $500\times 400$, and then the $360\times 288$ resolution images obtained by center cropping are used as training data.

To evaluate the performance for image content preservation at each level, we conduct experiments on three vision tasks: semantic segmentation, object detection, and edge preservation.

Intersection-over-Union (IoU) [49] is a widely used metric in semantic segmentation tasks. The mean value of IoU for all classes, denoted as mIoU, is adopted to evaluate the semantic consistency of NTIR image colorization methods.

Average precision (AP) [50] denotes the average detection precision of the object detection model under different recalls. The mean value of AP for all categories, defined as mAP, is selected as an overall evaluation metric.

APCE [9] is the average precision of Canny edges under multi-threshold conditions, and is employed to evaluate the edge preservation performance of the NTIR2DC model.

The translation results and corresponding semantic outputs of various methods are shown in Fig. 5. Column (a) represents the reference nighttime visible color (NVC) image and its semantic segmentation. The segmentation model fails to discriminate the pedestrians on the side of the car due to the bright beam of car headlights and low surrounding illumination. As shown in the white dashed boxes in the second row, all the compared I2I translation methods fail to generate plausible pedestrians. In contrast, the proposed model can maintain the complete pedestrian region and pose to facilitate the segmentation model’s discrimination, whether it covers a crowd (e.g., left side of the road) or an isolated pedestrian (e.g., right side of the road). Furthermore, the proposed approach outperforms other compared approaches for the structural preservation of the unlabeled sidewalk category (i.e., the pink region in the semantic mask).

Table I reports the quantitative comparison of the semantic consistency of various translation models. The proposed MornGAN outperforms other methods in terms of semantic retention in four large sample categories (i.e., road, building, sky, and car) and three small sample categories (i.e., traffic sign, pedestrian, and motorcycle). As there are few available samples and diverse colors, all methods have poor translation performance for the traffic sign, truck, bus, and motorcycle categories. Benefiting from a memory-guided collaborative attention strategy, the proposed method slightly outperforms other methods in semantic preservation for traffic signs and motorcycles. However, due to the lack of spatial continuity constraints on large object representation, the proposed method produces limited improvement in translation performance for truck and bus. Overall, the proposed method outperforms the other methods by a significant margin (i.e., at least 8.2%) in terms of mIoU for scene layout maintenance.

Better translation should facilitate better object detection. Fig. 6 presents the qualitative comparison of the colorization and object detection by YOLOv4 [52] on the translated images of various I2I translation methods. As shown in the red dashed boxes, almost all methods fail to reasonably translate the distant car except ours, which demonstrates the superiority of the proposed method for small object preservation. For the translation of the occluded objects, as shown in the green dashed boxes, all the I2I translation methods fail to make YOLOv4 recognize the complete six pedestrians except ours. Although YOLOv4 can identify pedestrians in well-lit areas of the NVC image, it cannot identify pedestrians in low-light areas (i.e., the area between the red dashed box and the green dashed box in the original image), which can be complemented by the proposed method.

As the bounding-box annotation of the FLIR dataset covers only three categories (i.e., pedestrian, bicycle, and car), quantitative comparison of the various colorization results for object detection is shown in Table II. The proposed method outperforms the other methods by a clear margin in object preservation for all categories. For example, MornGAN outperforms the second-ranked PearlGAN by a significant margin of 13.1%, which indicates the superiority of our method in object retention.

Fig. 7 visually compares the edge consistency of various translation methods. As shown in the blue dashed boxes, the edges of the buildings in the results of CycleGAN, ToDayGAN and DRIT++ are outwardly expanded, while the edges of the trees in the other four compared methods are inwardly shrunken. On the contrary, our model provides a complete match to the edges of the original image. In addition, as shown in the orange dashed boxes, ForkGAN, PearlGAN, and DlamGAN fail to maintain the continuous structure of the pole, and the edges of the pole are inwardly contracted in the other four methods. Compared with the other methods, MornGAN can more faithfully adhere to the edge structure of the original image.

As the Canny edges in the Fig. 7 are only the results of a fixed threshold, we exploit the APCE metric, which covers multiple thresholds to comprehensively evaluate the edge consistency performance, as shown in Fig. 8(a). We can find that the proposed method significantly outperforms other methods in edge consistency at all thresholds and is far superior to the second ranked DlamGAN by 18%.

Fig. 9 visually compares the colorization results of various translation methods on the KAIST dataset and their segmentation outputs. The relatively reasonable segmentation results in column (a) demonstrate the applicability of the selected segmentation model proposed in [34]. However, as shown in the white dashed boxes, all the compared I2I translation methods are unable to generate realistic pedestrians for the segmentation model to discriminate. Instead, the proposed method not only maintains most of the semantics of the dashed box region but also provides partial clues for distant pedestrian detection.

Further, a quantitative comparison of the semantic preservation performance of various I2I translation methods is shown in Table III. Despite the poor image quality, which makes scene understanding extremely difficult, MornGAN achieves the best performance among all methods in terms of semantic preservation for each category. Similar to the results on the FLIR dataset, all I2I translation methods have poor semantic retention in small sample categories. With the help of the proposed learning approach, features of small sample categories can be better retained compared with other methods. Overall, the semantic consistency of MornGAN on the NTIR2DC task is far superior to that of other I2I translation methods, that is, at least 7.1% ahead.

For pedestrian preservation, a qualitative comparison of various translation methods on the KAIST dataset is shown in Fig. 10. As shown in the two dashed boxes, all the methods cannot maintain realistic and complete pedestrian features to make the detection model convincing. However, MornGAN can generate plausible features for near pedestrians while maintaining a relatively reasonable local feature distribution for distant pedestrians.

Further, the quantitative comparison for pedestrian preservation is shown in Table IV. Due to the large number of pedestrians and good lighting conditions in city scenes, the performance of pedestrian detection in the reference NVC images is slightly better than that of the proposed method. Nevertheless, MornGAN outperforms other methods both in terms of detection precision and recall, and has substantial advantages in terms of overall mAP metrics.

Fig. 11 qualitatively compares the edge consistency of various translation methods on the KAIST dataset. Column (b) shows the enhanced NTIR image overlapped with Canny edges. As shown in the orange dashed boxes, the streetlights are vanishing in the results of ToDayGAN and DRIT++, while the edges of the streetlights are indented in the results of UNIT, ForkGAN, and PearlGAN. The neighboring structures of streetlights in CycleGAN and DlamGAN deviate severely from the input NTIR image. Similarly, as shown in the blue dashed boxes, the structures of the poles in the results of PearlGAN and DlamGAN are disconnected, while the streetlights and their neighboring edges of other methods differ significantly from the original image. Instead, our results match well with the original image on the edges.

Furthermore, the edge consistency comparison under the multi-threshold condition is shown in Fig. 8 (b). Considering all the thresholds, the proposed method still exhibits high performance in the edge consistency of the translation.

In order to explore the generalization ability of various I2I translation methods to out-of-domain distributions, we apply each model trained on the KAIST dataset to the FLIR dataset abbreviated as K$\rightarrow$F and vice versa as F$\rightarrow$K. The results are shown in Table VI. For the performance of semantic preservation, we can find that the proposed method obtains the best mIoU among all methods for both K$\rightarrow$F and F$\rightarrow$K experiments. Similarly, for the comparison of edge consistency during translation, MornGAN outperforms other compared methods by a significant margin for APCE in both experimental paradigms. In summary, the results demonstrate the strong generalization capability of the proposed method for domain shift.

The effect of the cluster number in ACA loss on the colorization performance is shown in Table VII. When the number of clusters is small, the features of small sample objects are difficult to capture comprehensively, so the segmentation model fail to understand the presence of objects. However, when the number of clusters is too large, many irrelevant features in the objects may disturb the recognition of the segmentation model. Compared with semantic consistency, edge consistency during translation is less sensitive to the cluster number. Therefore, we set the cluster number to four in all experiments.

Fig. 13 shows four failure cases of MornGAN, where the input images in the first two columns are from the FLIR dataset, and the remaining images are from the KAIST dataset. As shown in the first and third columns, local areas of large scale pedestrians and cars are incorrectly translated as roads due to their similarity in temperature. Moreover, the model fails in generating plausible objects when the objects are close to the camera, as shown in the second and fourth columns. Therefore, more attempts should be made in the future to design reasonable modules to capture complete object representations, such as combining the integrity and continuity of Gestalt laws.