WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding

Video Captioning: TACoS [31] offers fine-grained cooking activities, while the MSVD [6], MPII-MD [32], and M-VAD [30] datasets present a broad open domain with a substantial volume of clips. Although the MSR-VTT [41] dataset is rich in movie scene captions and provides a fundamental scene-based approach, it lacks the specificity required for fine-grained descriptions. The Charades [35] and Charades-Ego [34] datasets contribute valuable insights into daily indoor activities with lengthy captions. The ActivityNet Captions [9] dataset broadens the domain of instance-based activities as a dense captioning task with a significant number of clips, but it does not match the level of detail in pedestrian behavior that WTS offers. The large-scale instructional dataset HowTo100M [25] encompasses a vast array of activities, but it provides limited length in caption information. TRECVID-VTT’20 [2] offers a noteworthy volume of open domain clips, yet it does not approach the intricacy of pedestrian-vehicle interactions as WTS does. In the context of traffic-specific datasets, BDD-X [17] marks a significant step with its focus on traffic scenes and considerable annotation detail for driver action explanation. However, WTS surpasses it with higher granularity in pedestrian behavior analysis and a larger volume of clips and annotations focusing on pedestrians. Notably, WTS is pioneering in its inclusion of 3D gaze data, providing unparalleled insights into pedestrian attention and behavior in traffic scenarios.

Video and image captioning are fundamental tasks in video understanding. Vid2Seq [43] introduces a model that integrates special time tokens in a language model to predict event boundaries and textual descriptions in the same sequence. T. Wang et al. [38] present PDVC, a framework for dense video captioning that uses parallel decoding and treats dense caption generation as a set prediction task. MPLUG-2 [40] leverages large-scale pre-training for a deep understanding of complex visual-language interactions. VALOR [7] is a framework for object retrieval tasks involving video and language input, excelling in processing complex queries and locating items based on descriptions. Recent foundation models have significantly improved performance in video-to-text tasks due to prior knowledge alignment. DriveGPT-4 [42], based on the GPT-4 architecture, integrates visual data and contextual understanding for autonomous driving scenarios, showing strong performance on the BDD-X [17] dataset. Caption Anything [37] generates accurate, context-aware captions for a wide range of video content, leveraging the segment anything model for descriptive and relevant captions. Video-LLaMA [47] combines linguistic, visual, and audio data for comprehensive video content understanding. We benchmarked recent Video LLM-based methods’ performance on the WTS dataset to evaluate their potential for fine-grained video-to-text tasks.

Video/image caption evaluation metrics including, reference-based ones such as BLUE[29], ROUGE[22], CIDER[33], METEOR[5], SPICE[1] and Rankgen[19]. However, due to the above methods focused on the word level similarity and its order, for the long caption, especially its semantic meaning evaluation is relatively difficult to judge that two paragraphs represent the same thing but different words. Recently, for video-language understanding benchmark, GPT-based metrics [3, 11, 20] have been developed for use, which are better for aligning the semantic meaning for evaluation. The main difference between our proposed LLMScorer and the above ones is LLMScore is designed with a customizable specified aspect considering the semantic meaning as well as syntactic structure similarity for a holistic caption correctness evaluation.

Camera views: Our recording setup includes three types of cameras: overhead, driver recorder, and ego-centric cameras. The multi-view setup is designed for vehicle-infrastructure cooperation, such as in smart cities, enhancing the accuracy of fine-grained descriptions and improving AD system safety features. It also helps avoid false negatives from vehicle blind spots and offers promising avenues for future research. We selected $18$ out of $24$ overhead views after removing occluded viewpoints. Each view records at $1080p$ resolution and $24$ fps, with calibrated camera parameters. The driver recorder uses a GoPro Hero10 with linear model settings at $1080p$ and $24$ fps. The ego-centric camera, Tobii Pro Glasses 3, captures $720p$ resolution videos at $24$ fps, providing accurate 2D gaze ground truth. Sample views of these cameras are illustrated in Figure 1.

Subjects: There are total $14$ subjects who joined the recording with $7$ females and $7$ males, whose ages are ranged from $16{\sim}50$ years old.

Scenarios: We follow the ISO34502 standard, a scenario-based safety evaluation framework used for automated driving systems as well as our scenario guideline. We created at least one scenario for each of the $138$ pedestrian-vehicle relative position patterns defined in it to construct the recording scenario. The scenario patterns and its recorded video sample frames are shown in Figure 4(a).

Captions: We provide fine-grained pedestrian-related behavior captions for each video segment in the traffic scenarios. It starts from the phase segmentation to find each behavior phase temporal localization part, then moves to describe the event in the segment into text along the temporal directions. The features of this caption can be drawn as 1. long paragraph; 2. fine-grained observation regarding the position, action, attributes, and attention of pedestrians and vehicles with the surrounding context; 3. focus on the target objective.

3D Gaze: 3D Gaze is provided for the target pedestrians as extra data for further use, such as using it as a prior for traffic accident analysis. We provide both 2D and 3D gaze annotations for the corresponding frames with the target pedestrian. Figure 1 shows the example of 2D/3D gaze annotations in the frame. Except for the above-ground truth, we also provide the 3D head position and raw 2D gaze ground truth acquired from the Tobii glasses. To check and visualize the 3D information appropriately, a 2D paired 3D scanned map is given.

Generally, collecting natural traffic events in the real world is challenge. WTS ensures a controlled environment for collecting traffic events, adhering to ISO34502 scenarios and using pre-defined context settings based on actual accident videos. All accident-related scenarios are staged by professional stuntmen to replicate real-world conditions accurately.

Recording environment: We use a driving school with several intersections and single roads as our recording environment. It is a $72\times 84$ meters outdoor space in total, and $15\times 15$ meters for intersection and $11m$ as width and $81m$ as length for the single road. We have installed 6 multi-lens (4 lenses) overhead view cameras BOSCH FLEXIDOME multi 7000i in our recording place, and each overhead view camera is attached to the electric pole around the road. Figure 4(b) shows the map of our recording place with the camera placement positions.

Recording process: A series of scenarios was listed on a worksheet. Random walk or standby action was performed by subjects before and after each event. Each event includes each phase from Pre-recognition to Avoidance. The traffic light in the intersection is operated normally without any pre-defined behavior. The events occurred at various positions to ensure the diversity of the dataset. In each event, target pedestrians will be involved in the scenario, and non-target pedestrians who only performed the random walk in the video to make the task setting close to the real-world setup.

Synchronization: Synchronizing multiple, heterogeneous videos is challenging, especially under unconstrained outdoor environments, where we have multiple dynamic cameras. We utilize the audio input that most commodity video cameras are equipped to synchronize the videos. We use analog radio signals as the audio syncing signals. The average sync delay is $0.015$ seconds, well below our frame interval of $24$ FPS. For quality assurance, human annotators checked for audio delays (echoing when two videos are out of sync by more than $0.03$ seconds) and labeled such videos for human modification.

Annotation of the detailed description of traffic video is not easy to ensure accuracy and bias from the human. To resolve this kind of challenge and provide high-quality annotations for each data, we introduce several novel manners for the annotation.

Captions: For the traffic event-related caption, two challenge points are: 1) the caption information requires professional knowledge and viewpoint that is hard to create by a normal annotator correctly. 2) to give the temporal localization information to each segment and create the related temporal-spatial behavior caption is still based on the intuition of the analyst’s experience, which is biased. 3) writing a long description of the video from humans requires a high concentration which is error-prone. To remove the bias and make the annotator perform the annotation without deep knowledge of the traffic safety analysis area, we introduced a semi-auto caption generation manner with the following flow as shown in Figure 5: 1. We cooperated with the professional analyst in the insurance company to regularize and unify the guidelines for the segment temporal localization manner. Annotators are asked to do the phase segmentation according to the guidelines. 2. We make structured knowledge from the existing traffic event textual description data. The structured knowledge will be a breakdown of over 180 factual items as a checklist related to the environment context, attributes, position, action, and attention from target pedestrians and vehicles. For the position (left, right, front, etc.) items regarding the pedestrian and vehicle, we use the vehicle-centric as the relative anchor to define whether the pedestrian is "left" or right" to the vehicle to remove the bias. Samples of our checklist are shown in Figure 5. Details can be referred from the supplementary. 3. Annotators are asked to check the items from the structured checklist for each annotated segment according to facts that occurred in the video. 4. Once the event check process is done, a double-check process happens to verify whether the checked items match the video or not as the first quality verification. 5. Then, the checked items are fed to a Large Language Model with an appropriate prompt setting (detailed can be referred from the supplementary) for generating the natural sentence, including all the checked items as the caption ground truth. We use GPT-3.5 [28] as the LLM engine for the caption generation. 6. Finally, a double-check for the generated caption is done manually as the second quality verification.

Notice that GPT is used solely to summarize human-annotated scenario-describing checklist results into captions, ensuring that the diversity of scenario descriptions is not limited by GPT’s diversity.

3D Gaze: Previous studies on 3D gaze ground truth have typically been conducted in controlled environments or with people under controlled conditions. These studies often rely on numerous AR markers to localize ego-view cameras [26] or restrict the point of gaze, providing instructions to subjects for gaze estimation [16, 14]. However, these methods can make the environment appear unnatural in third-person view videos, or hinder subjects from acting naturally.

To overcome these limitations, our approach involves localizing in the environment through structure-from-motion (SfM) from ego-view video, inspired by [12]. The pipeline is outlined in Figure 6. Input videos are sub-sampled at 5 fps, and each frame is localized in world coordinates using SfM and a pre-built localization map. We utilize the GTSfM open source SfM library [4] and a Matterport camera with LiDAR scanning ¹¹1https://matterport.com/ to create the pre-built localization map. Finally, the 3D gaze direction in ego-view is transformed into each surveillance camera’s third-person view. This is done using the pre-calibrated surveillance camera pose, the ego-view frame pose, and the 3D gaze direction from the Tobii glasses in local coordinates. The transformation process for the $i$-th surveillance camera is defined as:$\mathbf{d}_{i}=\mathbf{R}_{i}^{-1}\mathbf{R}_{\text{ego}}\mathbf{d}_{\text{ego}}$, where $\mathbf{d}_{\text{ego}}\in\mathbb{S}^{2}$ represents the 3D gaze direction in ego-view, $\mathbf{R}_{\text{ego}}\in\text{SO}(3)$ the ego-view pose in world coordinates, $\mathbf{R}_{i}\in\text{SO}(3)$ the $i$-th surveillance camera pose in world coordinates, and $\mathbf{d}_{i}\in\mathbb{S}^{2}$ the 3D gaze direction in the $i$-th surveillance camera.

We evaluated the 3D gaze annotation quality in four aspects: (1) ego-view pose estimation using SfM, (2) transformation to world coordinates, (3) overhead camera pose estimation, and (4) Tobii’s accuracy with moving subjects. For (1), sampled videos with rapid motion were used, resulting in an average error of $4.19$ degrees. For (2) and (3), we estimated relative poses between real and rendered images from scanned 3D scenes, achieving an error of $0.18$ degrees. We applied a sanity check to remove visually perceptible errors and aggregated ego-view poses using RANSAC and Procrustes. For (4), Tobii’s gaze accuracy for walking subjects was $1.74$ degrees [27]. Finally, the combined error was $6.11$ degrees, within the human eye’s $15$ degree eccentric gaze FOV.

Bounding Box Generation We also provide bounding boxes of target vehicles and pedestrians. We choose a semi-supervised approach based on the human prompt in the first frame and tracking the target in the rest of the frames. We leverage Track Anything [44], an interactive tool for segmentation and video object tracking based on Segment Anything [18] and XMem [8] respectively, which only takes several clicks on the target in the first frame as input.

Dataset for benchmarking: There are around 120 scenarios from staged data and 2000 scenarios from selected BDD data as training sets, 60 scenarios from staged, and 800 scenarios from BDD as validation sets. Each scenario will have $\sim 5$ segments with $\sim 5$ captions. For staged data, as there are multiple overhead views, we picked up a main view that covers the whole scenario in one view with a clear visible angle for the pedestrian and vehicle from the dataset for training and validation purposes. We think multiple-view caption consistency is also a new aspect for pushing forward more accurate video-to-text performance.

LLMScore Evaluation protocol: Semantic and syntactic information is crucial to measure the relatedness of two sentences. Various studies [15] [10] learn to disentangle the semantic and syntactic representations. To achieve this, inspired by the evaluation protocol in GPTScore [11], our LLMScore has a prompt template includes task description (comparing two captions), ground truth caption ($G$), inferred caption ($C$), and consideration aspects (location, attention, behavior of pedestrian/vehicle, and environment). Semantic score quantifies the degree of semantic similarity between two sentences. In our approach, we instruct the LLM (GPT-3.5-turbo) to evaluate the semantic accuracy of the caption for each aspect Location, Attention, Behavior, Context with respect to the ground truth. We assign a score of $1$ to the aspect that is semantically correct, otherwise $0$. We take an equal-weighted average of these scores and call this Semantic Score ($Score_{sem}$). Syntactic score quantifies the syntactic similarity between the answers. First, for each aspect, we prompt LLM (GPT-3.5-turbo) to give the answers to subjective questions for candidate caption as well as ground truth. Subsequently, we compute the cosine similarity score between the embeddings of these answers for the subject questions associated with candidate caption and ground truth. We use OpenAI’s text-embedding-3-small for generating embeddings. We then calculate the syntactic score ($Score_{syn}$) by taking the equally weighted average of the cosine similarities. Semantic score and Syntactic score are defined as $Score_{sem}$ and $Score_{syn}$ respectively as,

where $P_{sem}$ and $P_{syn}$ are the prompt template with the input, $T$ is task definition, $A$ is definition for aspects, $G$ is ground truth, $C$ is inferred caption. $Q_{sem}$ defines the queries, $S_{sem}$ is scoring criteria, $O_{sem}$ is output format for semantic scoring and $Q_{syn}$ defines the queries, $S_{syn}$ is scoring criteria, $O_{syn}$ is output format for syntactic scoring. Therefore, LLMScore is defined as $LLMScore=w_{1}*Score_{sem}+w_{2}*Score_{syn}$, where $w_{1}$ and $w_{2}$ are the weights for the scores.

For the Video-LLaMA and Video-ChatGPT, we did not fine-tune the model but used the input video and the user query prompt fed to the LLM for generating the captions. There are three kinds of prompts we used, P-A is a simple task prompt like "Describe the video from a pedestrian perspective". P-B is a prompt with system settings and more constraints for the traffic domain for the task. P-C is a system prompt with a task description following a demonstration sample. More detail can be referred from the supplementary. For our proposed baseline, during the fine-tuning, LLM is frozen only to fine-tune the Q-Former, linear translation part. We use the AdamW optimizer and a weight decay of $0.05$. We use a cosine learning rate decay with a peak learning rate of $2e-5$ and a linear warmup of $2k$ steps. We use images of size $22\times 224$. For LLMScore, the $w_{1}$ and $w_{2}$ are set to $0.7$ and $0.3$ respectively.

Table 2 shows the comparison result on the WTS dataset for each method with different prompt settings on 4 kinds of traditional popular metrics. It is obvious that the prompt $P-C$ is the best setup and thus all the methods could achieve the best results under this setting. However, based on this best prompt setting, Video-LLaMA and Video-ChatGPT still worked not well for the WTS dataset traffic event domain showing that the fine-grained description in WTS is not generalizable from the common sense knowledge trained Video LLM model without fine-tuning.

To evaluate the impact of fine-tuning on the WTS dataset, we compared Ours(VideoLLM) to Video-LLaMA in Table 2 of our paper. Both models share similar architectures, isolating the effect of fine-tuning. The comparison showed that fine-tuning improves performance but is still a challenge. In Ours(Instance-VideoLLM), we added region-specific information. Despite this, results indicate significant room for improvement, suggesting our approach as an initial idea for developing more advanced methods for fine-grained instance-level video understanding with LLMs.

Table 3 shows the result using our LLMScore for each method. For Video-LLaMA and Video-ChatGPT, the semantic score is relatively low and the syntactic score is high, is because almost all the critical semantic meaning regarding the location, attention, behavior, and context are not correct according to the GT even though the whole paragraph looks similar to each other. More samples can be referred from the supplementary. Figure 9 shows a success case sample of how the caption looks like from each method. It is hard to tell the difference from the traditional metrics but highly be separated by using the LLMScore for fine-tuned results, as more semantic meanings are correct for this case.

To compare the LLMScore with the human evaluation result, We use 50% of the validation set for human evaluations. Human evaluators score the correctness of information in $C$ and $G$ using pre-set questions, extracting sub-texts that best describe the aspects. The average human score is $0.243$ (variance is $0.002$), close to the LLMScore of $0.242$ (variance is $0.0007$) for the same samples.