Intersection Prediction from Single $360^{\circ}$ Image via Deep Detection of Possible Direction of Travel

Movie-Map [Lippman(1980)] was proposed in 1980 as the first interactive map to engage a user in a simulated driving experience. The original Movie-Map system was built using an optical videodisc and four stop-frame film cameras – the cameras, mounted on the top of a car, were triggered approximately every 10 ft. Movie-Map simulates travel by displaying controlled rate sequences of individual frames captured at periodic intervals along a particular street in a town. Despite being an innovative concept, the system was impractical because of the large human efforts involved. For instance, to allow the route to deviate from straight paths down each street, separate sequences were captured to display all the possible turns at every intersection. In addition, the captured videos had to be split manually by intersection to connect the different driving videos through those turn sequences. Owing to the lack of scalability caused by the large human effort as well as that of computational resources and data capacity at that time, Movie-Map was relegated to less importance until recently, and a system based on static $360^{\circ}$ images (such as GSV [Google(2005b)]) became the mainstream approach to digital map navigation.

Recently, Sugimoto et al\bmvaOneDot  redesigned Movie-Map with modern imaging and information processing technologies and demonstrated its superiority to GSV in exploring unfamiliar scenes [Sugimoto et al.(2019)Sugimoto, Iinuma, and Aizawa, Sugimoto et al.(2020b)Sugimoto, Okubo, and Aizawa]. In addition to replacing large-camera and expensive disk systems in [Lippman(1980)] with consumer $360^{\circ}$ cameras and personal computers, they proposed a method to automate the time-consuming intersection segmentation. Specifically, they applied Visual SLAM [Lothe et al.(2009)Lothe, Bourgeois, Dekeyser, Royer, and Dhome] to recover and track the 3-D camera trajectories of the walk-around videos, aligned the trajectories onto the map, identified the intersections using the aligned trajectories, and refined them using visual features. However, their method relied heavily on the results of Visual SLAM, which is not robust to dynamic objects such as cars or people and texture-less landscapes. On the other hand, as our method does not rely on either 3-D reconstruction or multiple frames from the entire frame sequence, it is less sensitive to dynamic objects and low-textured scenes.

In addition to Movie-Map, intersection identification using a pedestrian viewpoint or vehicle-mounted cameras has recently been actively studied in automated driving and robot navigation fields. Owing to the highly unpredictable behavior of traffic intersections, correct recognition and safe behavior in their proximity is essential for an autonomous agent.

Depending on the input information, intersection identification methods are broadly categorized into two types: those that use non-visual information such as LIDAR or GPS data and those that use images or videos. Without using visual information, Fathi and Krumm [Fathi and Krumm(2010)] identified intersections in an urban-scale traffic network using GPS data from several vehicles. On the other hand, Zhue et al\bmvaOneDot [Zhu et al.(2012)Zhu, Chen, Li, Li, Nüchter, and Wang] proposed a method for intersection detection using sparse 3-D point clouds obtained from in-vehicle 3-D LIDAR data. Unfortunately, GPS data are frequently inaccurate in urban areas with tall buildings, and 3-D LIDAR is expensive and basically less portable than consumer cameras, which are inappropriate for capturing walk-around videos for some applications such as Movie-Map.

With the development of deep learning technologies, image-based methods for intersection identification are becoming more attractive owing to their cheap installation cost and compatibility with in-vehicle or robot-mounted cameras. Bhatt et al\bmvaOneDot [Bhatt et al.(2017)Bhatt, Sodhi, Pal, Balasubramanian, and Krishna] formulated this task as a binary classification problem to identify intersections in short-time on-board videos and developed a variant of long-term recurrent convolutional network as the classifier. Although the classifier has been shown to be effective for in-vehicle videos, its reliability in other domains such as walk-around pedestrian-view videos, is still unclear. In addition, as this model accepts multiple video frames as input, its robustness in dynamic scenes with several walking people, such as in urban shopping malls, is questionable. On the other hand, Astrid et al\bmvaOneDot [Astrid et al.(2020)Astrid, Lee, Zaigham Zaheer, Lee, and Lee] recently proposed a single-image intersection classification system using ResNet-based architecture trained on a pedestrian-view-level image dataset containing 345 and 498 intersection and non-intersection images, respectively. Although the system achieved a test accuracy of 80%, the training and test data comprised pedestrian-view images of sidewalks with little or no pedestrian traffic, and the generalization of this method to disparate test scenes from the training dataset is still not confirmed.

Unlike all these studies, the input of our method is a single frame from $360^{\circ}$ walk-around videos recorded for Movie-Map. To the best of our knowledge, this is the first attempt to identify an intersection from a single $360^{\circ}$ image. In addition, our targets include not only in-vehicle images or places with no pedestrians but also various cities, towns and villages with active people and car traffic, including places with different building designs. In such cases, conventional approaches such as direct binary classification of images are difficult to implement. Therefore, we propose a new intersection identification method based on the detection of PDoTs.

The maximum number of Movie-Map videos have been shot by a photographer walking on the sidewalk in a wide street. Therefore, the intersection detector should be trained in the same domain, rather than on videos shot by car-mounted cameras such as in GSV [Google(2005b)]. However, as the diversity of the intersections in the two areas mentioned previously was insufficient for generalizing the intersection identification, we supplemented it by adding another training dataset, Suburb-GSV images. The added area contains well-maintained streets but scarce traffic lights and signs; moreover, the dataset contains a few scenes without any walls or buildings and low textures. We collected 500 suburb intersection $360^{\circ}$ image from GSV and used them for training in addition to the images from Campus and Downtown.

Next, for each key interaction frame included in this frame set, PDoTs were manually annotated on a user interface. Specifically, a worker was asked to indicate the PDoT in a frame with icons specifying a single or ”omnidirectional” PDoT, where all the directions are assumed to be PDoT, for example, in a plaza on the university campus. After completing all the annotations, we cropped the NFoV (45°in horizontal and vertical views) images approximately centered at PDoT from the $360^{\circ}$ image as positive examples. It should be noted that, to inject noises in the training examples, we did not crop the NFoV image strictly centered at the PDoT and randomly shifted it up to 5 degrees. After the extraction of the positive examples from a single frame, the negative examples were similarly extracted from the same frame as NFoV images centered between two adjacent PDoTs. Notably, this sampling procedure can be applied to both key intersection and non-intersection frames (i.e., non-intersection frames have their PDoTs in the forward and backward directions). As shown in Fig. 5, this procedure basically generates slightly more positive than negative examples because a few negative candidates are discarded when the inner angle between two adjacent PDoTs is less than 45°(e.g\bmvaOneDot, Y-junction; otherwise, the negative example must contain PDoT). To equalize the number of positive and negative examples, we added additional negative examples from other random regions. Consequently, we obtained 3414 positive examples (i.e., 907 in Campus, 971 in Downtown, and 1536 in Suburb) and 3274 negative examples (859 in Campus, 882 in Downtown, and 1510 in Suburb). These samples were precisely labeled and diverse in terms of landscape.

Specifically, equipped with the annotation of key intersections, we re-assigned the soft intersection label to every 10 frames in the videos based on the frame positions from the key intersection frame. We shot walk-around videos in Suburb similar to that in Campus and Downtown areas. Frames within $0.5$ s walking distance (e.g\bmvaOneDot, $15$ frames in the $30$-fps video) from the key intersection frame were labeled as one (positive) and frames with more than two second walking distance as zero (negative); the frames decreased linearly from one to zero between $0.5$ s and two second walking distances. Because a majority of the frames were negative examples (i.e., far more than two second walking distance from the key intersection frame therefore labeled as “False”), we balanced the portions of positive and negative examples by only extracting $p$ percentage of negative ones. We empirically found that $p=20\%$ is the optimal percentage, which means that 80 percentage of negative samples were discarded. Finally, we obtained 544 positive examples, 1225 soft-labeled examples (i.e., labeled between zero and one), and 8159 negative examples. The total distribution of (positive/soft-labeled/negative) examples for each area was (198/420/2188) for Campus, (185/457/3656) for Downtown, and (161/348/2315) for Suburb. Notable, The labels were treated as a continuous probability distribution.

The same test task was employed for both the PDoT-based and näive direct approaches: intersection identification from a single 360°image. We prepared $50$ intersection and non-intersection frames, respectively, for each of the three areas and added a completely new scene called Chinatown to validate the ability of the network to generalize to unknown scenes. The test data resulted in a total of $400$ frames. Notably, all the test images were not included in our training data.

Our method and Baseline were implemented using the PyTorch framework [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala]. The backbone architecture for both the methods was Resnet-50 [He et al.(2016)He, Zhang, Ren, and Sun], pre-trained on ImageNet [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, and Fei-Fei] except for the final classification layers and fine-tuned on the iii360 dataset. This implies that the only difference between these two networks was the input and output information. The input for our method was an NFoV image cropped from a target $360^{\circ}$ image, whereas that for the Baseline was the $360^{\circ}$ target image itself. The output was the binary labels w.r.t PDoT or intersection. Both networks accepted images resized to $224\times 224$ as input and were trained and tested on a machine with single NVIDIA TITAN Xp with 12GB of GPU memory. For optimization, we used Adam [Kingma and Ba(2014)] with a learning rate of 0.001 and batch size of $8$. The number of epochs was set to $300$ for all network and training set combinations. For data augmentation, horizontal flipping, color jitter, and random erasing were applied.

To identify the intersection, our method merges the multiple PDoT predictions as follows. First, in a provided target $360^{\circ}$ image, horizontally non-overlapping eight perspective images of $45^{\circ}\times 45^{\circ}$ FoV are cropped (in a random start direction). Then, each NFoV image is individually fed to the PDoT prediction network, and the target $360^{\circ}$ image is identified as an intersection if when more than three PDoT predictions are positive.

We investigated the effect of the number of perspective images cropped from 360° images. We varied it within 8, 16 and 32 (i.e., 45° , 22.5° and 11.25° FoV, respectively) and compared the prediction performance by training the network on Campus and testing on Chinatown datasets. The result is shown in Table 3, and we found the result of 8 views is the best.

In addition, in order to evaluate the effect of the resolution of input images in the direct method, we compared the prediction accuracy by training the network on Campus and testing on Chinatown datasets by varying the image size among $224\times 224$, $448\times 448$ and $896\times 896$.  Table 3 demonstrated that the difference of input image size is not influential to the performance.

We compared our method with Baseline having different combinations of training/test data. In addition to training the networks on examples from individual areas, we attempted training on all the examples from all areas. Notably, we did not evaluate PDoT prediction accuracy because this was not our main task; instead, we assigned more importance to the intersection identification performance.

The comparative intersection identification accuracy results are shown in Table 1. It is evident that the prediction accuracy of our method is better than that of Baseline for all the combinations of training and test areas, indicating that our PDoT-based algorithm consistently outperforms the näive direct approach of the Baseline algorithm.

As expected, the performance was better when the training and test examples were drawn from the same area, compared to when drawn from different areas. Interestingly, the network trained on Suburb-GSV still performed satisfactorily on the test Suburb dataset despite the completely different device setups for obtaining both types of data, indicating that a difference in the image acquisition setup exerts a lesser effect than that in the area where the image was captured. The $0.87$ prediction accuracy demonstrated by the network trained on Downtown and tested on Chinatown also supports this observation as both these areas have several common characteristics, such as movement of people through narrow and intricate shopping streets. Conversely, the network trained on Campus did not perform well on Chinatown where both areas have less shared characteristics. Although the excellent performance of the network on Downtown even after training on Campus appears counter-intuitive, the average prediction accuracy on Downtown is consistently high and simply indicates that intersection identification in this area is relatively easier than in other areas with more diversity in the appearance of intersections. It is not surprising that the network trained on all the training examples performed slightly worse than when the training and test examples were drawn from the same area. Instead, this result indicates that the network is capable of covering a variety of areas by drawing training images of intersections from them.