Think Twice Before Recognizing:
Large Multimodal Models for General Fine-grained Traffic Sign Recognition

The early TSR studies focus on performing recognition based on hand-crafted features and machine learning algorithms. For example, hand-crafted features were used to extract features from traffic signs and machine learning algorithms recognized the extracted features. Zaklouta et al. [5] introduce a real-time system for detecting and classifying circular and triangular traffic signs. Kus et al. [41] introduced a method for detecting and recognizing traffic signs by improving upon the SIFT [8] algorithm. They enhanced SIFT by integrating features associated with the color of local regions. Huang et al. [7] propose a streamlined method for TSR by using HOG features and a single classifier trained with the extreme learning machine algorithm. The HOG feature strikes a balance between redundancy and local details, improving the representation of distinctive shapes. In this way, the traditional methods rely heavily on hand-crafted features, which are sensitive to variations in lighting, occlusion, and complex backgrounds [42].

The emergence of deep learning has inspired TSR research. Compared with traditional hand-crafted feature-based methods, deep learning-based methods can better learn features of traffic sign images. Zhang et al. [43] introduced two lightweight networks for improving recognition accuracy with fewer parameters. Abudhagir et al. [44] utilized the LeNet model for traffic sign recognition. Their CNN architecture consisted of the first two layers adapted from LeNet, followed by two additional convolutional layers, a dropout layer, and a flattened layer. Zhu et al. [45] proposed a TSR method based on YOLOv5. Besides, transformer-based TSR methods have also been proposed. Zheng et al. [18] used a vision transformer (ViT) [46] to perform a detailed TSR evaluation. Luo et al. [16] proposed a TSR approach comprising a lightweight pre-locator network and a refined classification network based on Swin-Transformer [47]. The pre-locator network identifies sub-regions of traffic signs, while the refined classification network handles recognition within these regions. Guo et al. [17] proposed an end-to-end framework that integrates component detection, content reasoning, and semantic description generation for understanding traffic signs. Since trained on country-specific datasets, these supervised methods require fine-tuning or training from scratch when recognizing traffic signs in other countries. Several TSR approaches have been introduced to solve the problem. Cao et al. [48] proposed a zero-shot method, which synthesizes a hybrid feature representation by extracting both general and principal visual features from traffic sign images. Gan et al. [23] introduced a zero-shot approach utilizing midlevel features extracted from CNNs. However, due to the existence of cross-domain biases and the need for improving accuracy, more effective methods are expected to be explored.

In the proposed method, we first perform segmentation of the original road image $\mathcal{I}_{o}^{i}$ containing the traffic signs $i\in\{0,1,2,\ldots,N\}$. $N$ represents the number of traffic signs contained in the original road image. The original road image $\mathcal{I}_{o}^{i}$ is input to the segmentation model. The segmentation model generates segmentation images $\mathcal{I}_{s}^{i}$ with various object category labels for the original image. As the recognition of traffic signs is performed, the traffic signs need to be distinguished from other objects. Specifically, in the segmentation image $\mathcal{I}_{s}^{i}$, each particular object category is coded as a different color for identification. We convert $\mathcal{I}_{s}^{i}$ to a mask image $\mathcal{I}_{m}^{i}$, thereby separating the traffic sign from the other objects and background in $\mathcal{I}_{s}^{i}$. The segmentation model is not limited to a specific architecture.

The extraction of traffic signs is realized by a designed extraction detector. The extraction detector first obtains the coordinates of the traffic signs in the mask image $\mathcal{I}_{m}^{i}$ using the contour detection algorithm  [54]. Then, the extraction detector uses the original road image $\mathcal{I}_{o}^{i}$ and the coordinates of the traffic signs to extract the image $\mathcal{I}_{r}^{N}$ that contains only the real traffic signs. $\mathcal{I}_{r}^{N}$ removes other objects and backgrounds in the original road image. The extraction detector finally retrieves the traffic sign image $\mathcal{I}^{i}$ from $\mathcal{I}_{r}^{N}$ using the corresponding coordinates of the traffic signs. $\mathcal{I}^{i}\in\mathbb{R}^{\mathcal{H}\times\mathcal{W}\times 3}$ represents the final extracted traffic sign image. Note that while $\mathcal{I}^{i}$ can also be obtained directly from the original road image $\mathcal{I}_{o}^{i}$ via the coordinates, the extracted traffic sign image contains unnecessary backgrounds. In contrast, the designed extraction detector can remove the backgrounds and avoid potential interference for subsequent recognition.

In the proposed method, prior knowledge includes context descriptions of original road images, characteristic descriptions of template traffic signs, and differential descriptions of similar traffic signs. The inputs of LMMs are typically an image $\mathcal{I}^{i}$ and a text query $\mathcal{T}^{i}=[t^{i}_{1},\ldots,t^{i}_{l_{i}}]$ with length $l_{i}$, and LMMs generate a sequence of textual output $\mathcal{T}_{\text{out}}^{i}=[t_{1}^{i},\ldots,t_{l_{o}}^{i}]$ with length $l_{o}$ as follows:

Context Descriptions: Since original road images contain important contextual information about traffic signs, we transform original road images into context descriptions to fully utilize the scene information. Given an original road image $\mathcal{I}_{o}^{i}$, the context descriptions $\mathcal{D}^{i}_{\text{Cont}}=[\mathcal{D}^{i}_{\text{Cont}},...,\mathcal{D}^{N}_{\text{Cont}}]$ are generated as

where $\mathcal{T}^{i}_{\text{Cont}}$ represents the prompt for generating the context descriptions. As shown in Fig. 3, we carefully designed $\mathcal{T}^{i}_{\text{Cont}}$ so that the generated contextual descriptions contain the context background information understood by LMM from the original road image. Furthermore, as in the real-world question-answering process, we find that narrowing the range of answers can reduce the recognition difficulty of LMM. To this end, we propose a prior traffic sign hypothesis, which allows LMM to filter irrelevant traffic sign types and provide potential candidates. Similar to human cognition, where irrelevant answers are swiftly filtered based on existing knowledge, the potential traffic sign candidates generated by the prior traffic sign hypothesis are obtained from LMM’s preliminary understanding of the original road image. This preliminary understanding stimulates subsequent detailed thinking. In addition, when multiple traffic signs exist in the original road image, it is difficult for the LMM to perform context description and the prior traffic sign hypotheses generation. Therefore, we simplify the complex and propose a prompt optimization method based on center coordinates. The prompt optimization method provides the center coordinates of traffic signs to inspire the LMM to locate the target traffic sign from the original road image. Since the center coordinates are obtained from the extraction detector, no additional calculations for center coordinates are required. The center coordinates help the LMM to locate the target traffic sign and generate corresponding background descriptions and prior traffic sign hypotheses.

Characteristic Descriptions: Since TSR is a fine-grained task, it is difficult for LMMs to accurately answer the specific types of traffic signs based on the existing knowledge. We consider that template traffic signs are easily accessible in national traffic sign databases, which helps to reduce the dependence on training data. Although previous TSR methods have utilized template traffic signs at the feature level, actual traffic sign images are diverse due to lighting conditions, angles, occlusions, etc., and can be different from template traffic sign images. It increases the difficulty of cross-domain recognition at the feature level. We introduce the few-shot in-context learning to generate characteristic descriptions $\mathcal{D}^{C}_{\text{Char}}=[\mathcal{D}^{1}_{\text{Char}},...,\mathcal{D}^{c}_{\text{Char}}]$ of each class $c$ of template traffic sign $\mathcal{I}^{C}_{\text{Temp}}=[\mathcal{I}^{1}_{\text{Temp}},...,\mathcal{I}^{c}_{\text{Temp}}]$ with the prompts $\mathcal{T}^{C}_{\text{Char}}=[\mathcal{T}^{1}_{\text{Char}},...,\mathcal{T}^{c}_{\text{Char}}]$ as follows:

where $\mathcal{T}_{\text{Char}}^{c}$ represents the corresponding prompt for the template traffic sign $\mathcal{I}^{c}_{\text{Temp}}$. As shown in Fig. 4, traffic signs in all countries have three key features: shape, color, and composition. When performing in-context learning, the designed prompt makes the LMM focus on these three features without introducing any other irrelevant information. The proposed in-context learning method only needs to generate the characteristic descriptions once for each template traffic sign. By avoiding computation at the feature level, the generated characteristic descriptions can reduce cross-domain differences between templates and real traffic signs. The prompts we design are uniform and simple for each traffic sign and require no special adjustments. We construct a memory bank to store the generated characteristic descriptions.

Differential Descriptions: Since the characteristics of certain types of traffic signs are similar, we generate differential descriptions to emphasize the differences between these traffic signs. Given a template traffic sign $\mathcal{I}^{u}_{\text{Temp}}$ and the similar traffic sign $\mathcal{I}^{v}_{\text{Temp}}\in[\mathcal{I}^{1}_{\text{Temp}},...,\mathcal{I}^{S}_{\text{Temp}}]$, the characteristic descriptions of $\mathcal{I}^{u}_{\text{Temp}}$ and $\mathcal{I}^{v}_{\text{Temp}}$ can be obtained by Eq. (3), and are respectively expressed as

The differential descriptions $\mathcal{D}^{(u,v)}_{\text{Diff}}$ are then obtained by inputting $\mathcal{D}^{u}_{\text{Char}}$ and $\mathcal{D}^{v}_{\text{Char}}$ into the LMM as follows:

where $\mathcal{T}^{(u,v)}_{\text{Diff}}$ represents the prompt for generating the differential descriptions of $\mathcal{I}^{u}_{\text{Temp}}$ and $\mathcal{I}^{v}_{\text{Temp}}$. Thus, the final differential descriptions $\mathcal{D}_{\text{Diff}}$ are expressed as

As shown in Fig. 5, experts perform the judgments of the signs similar to the traffic sign. The characteristic descriptions in the memory bank are imported to help generate differential descriptions of the traffic sign and each similar sign. The differential descriptions emphasize the subtle differences between similar traffic signs and can further optimize the fine-grained recognition ability of LMMs.

After obtaining the context descriptions $\mathcal{D}^{i}_{\text{Cont}}$, characteristic descriptions $\mathcal{D}^{C}_{\text{Char}}$, and differential descriptions $\mathcal{D}_{\text{Diff}}$, the LMM performs multistep reasoning for a target traffic sign. Step 1: LMM first conducts a preliminary understanding of the target traffic sign image based on existing knowledge. Step 2: LMM understands the scene information around the target traffic sign by referring to the context descriptions. Meanwhile, LMM further narrows the thinking scope by referring to the prior traffic sign hypotheses. Step 3: referring to the characteristic descriptions, LMM understands the basic features of various traffic signs of the shape, color, and composition, and compares the understanding of the target traffic sign image with the characteristic descriptions, thus stimulating the fine-grained TSR. Final: referring to the differential descriptions, LMM gains insight into understanding the differences between the target traffic sign and other similar traffic signs to optimize the recognition results as follows:

where $\mathcal{T}_{\text{Multi}}$ represents the designed multistep prompt, and $\mathcal{T}_{o}^{i}$ is the final TSR results of the LMM. Through multistep thinking, the LMM performs feature inference step by step to finally find out the “real face” of the target traffic sign. Multistep thinking can largely stimulate the LMM’s ability to recognize traffic signs at a fine-grained level. Therefore, the fine-grained TSR performance in real-world scenarios of LMM is improved.

To further verify the effectiveness of the proposed multi-step thinking strategy and explore the respective effectiveness of each proposed description. We calculated the Top-$k$ fine-grained TSR performance for different thinking strategies on five datasets as shown in Table II. When there are no context, characteristic, and differential descriptions, target traffic signs are directly input into LMM for recognition. Table II demonstrates that the results of baseline exhibit the lowest accuracy on all datasets compared with the performing of thinking strategy. As the thinking step increases, the Top-$k$ TSR recognition accuracy increases, which proves the effectiveness of the proposed method. Besides, the results also show that each proposed description has a positive impact on improving the fine-grained TSR capabilities of LMM. Comparing the results of using only one kind of description, the characteristic description contributes most to the TSR recognition accuracy. Through characteristic descriptions, LMM can consider the features of the target traffic sign and the descriptions of template traffic signs, thereby improving fine-grained recognition capabilities. In addition, context and differential descriptions show the ability to further optimize fine-grained TSR recognition on all five datasets, which is consistent with our hypothesis.

Figure 8 shows the recognition examples of the baseline and proposed method under five datasets (Gpt-4o). Compared to the baseline, our strategy shows stable recognition performance for traffic signs in real-world scenarios and can be generalized to recognize traffic signs in different countries. In particular, as shown in Fig. 8, most of the identified traffic signs of the baseline and our strategy show only minor differences. This illustrates that the proposed strategy helps the LMM to fully consider the diversity and similarity of traffic signs to perform accurate fine-grained level recognition. Figure 9 shows examples of recognition errors of the proposed method. When traffic signs are too blurred, it is difficult to understand the traffic sign images for accurate recognition.

To validate the effectiveness of the proposed prior traffic sign hypothesis and center coordinate prompt optimization method, we conducted experiments on the effectiveness of different context description generation methods on three real-world datasets, TT-100K, Sapporo, and Yokohama. Note that all results in Table III use context descriptions, characteristic descriptions, and difference descriptions for multi-step thinking. As shown in Table III, without the prior traffic sign hypothesis and center coordinate prompt optimization, i.e., with only simple image background descriptions in the contextual description, the Top-$k$ fine-grained TSR performance is reasonably similar to the accuracy in Table II using only the characteristic and differential descriptions. Since the characteristic and differential descriptions are performed, only simple background descriptions of images in the context description have limited contribution to improving the fine-grained capability of the LMM. The situation is also similar when only the center coordinates optimization is performed. Although the LMM can locate the target traffic sign from multiple traffic signs in the original road image and simply describe the features. However, the simple descriptions in the contextual description contribute limited since characteristic descriptions are already provided. When only the prior traffic sign hypothesis is performed without center coordinate prompt optimization, it is difficult for the LMM to locate the target traffic sign from multiple traffic signs in the original road image, thus generating confusing descriptions. The confusing descriptions negatively affect the accuracy. When both the prior traffic sign hypothesis and the center ordinate prompt optimization are performed, the Top-$k$ fine-grained TSR performance is improved by locating the target traffic signs and filtering irrelevant answers.

In Table IV, we validate the comparison of TSR performance under different thinking steps. For the GTSRB and BTSD datasets, we change the order of thinking for characteristic and differential descriptions. For the TT-100K, Sapporo, and Yokohama datasets, we change the thinking order of context and characteristic descriptions. The experimental results are shown in Table IV. After the order of thinking is changed, the TSR performance remains almost the same as the original, which shows the robustness of our method. Since the cues obtained by LMM are the same even when the order of thinking is changed, there is no significant difference in recognition accuracy.

Previous experiments demonstrate that the proposed multi-step thinking can be easily extended to different LMMs and maintains robust performance. In addition, in our designed traffic sign extraction module, the segmentation model is not limited to a specific model and can easily be extended to advanced models. Figure 10 shows traffic sign extraction examples with segment anything model 2 (SAM 2) [68] and ViT-Adapter [69]. As shown in Fig. 10, under different segmentation models, target traffic signs are extracted through the designed extraction module. The most advanced segmentation model such as SAM 2 performs better extraction on traffic signs.

Table V shows the inference speed of the proposed method. When the segmentation model is ViT-Adapter, the inference speed is 0.4s per road image, and the SAM 2-based extraction module can achieve real-time extraction. For LMM, the multistep thinking of Gpt-4o is faster than Gpt-4v by 0.4s and the fastest total inference speed is 1.2 seconds. Since our method can be extended to future LMMs, the inference speed has the potential to be faster.

In this work, we designed the differential descriptions for the LMMs and demonstrated the effectiveness of the differential descriptions. However, similar traffic signs are determined based on expert knowledge to generate the differential descriptions. In future work, automatic methods for determining similar traffic signs need to be explored.

In this paper, five datasets including three public datasets (GTSRB, BTSD, TT-100K), and two private datasets (Sapporo, Yokohama) are used to verify the performance of the proposed method. However, all five datasets were taken under sunny weather. As a result, the traffic sign images are relatively clear. Under the weather such as rain, fog, and snow), the traffic sign images can become blurred, which affects the TSR performance. Improving TSR performance in this condition is a direction we look forward to exploring in the future.