NYU-VPR: Long-Term Visual Place Recognition Benchmark
with View Direction and Data Anonymization Influences

Visual place recognition (VPR) is the process of retrieving the most similar images for a query one from a database of images with known camera poses, which is often used for loop closing in mapping, localization, and navigation. It relies on representing an image as a global feature vector which describes the portion of the image appearance that is most relevant to its capturing pose. Its applications range from autonomous driving for vehicles, to assistive navigation for the visually impaired people, especially in busy and crowded metropolitan areas where GPS could suffer from the “urban canyon” problem when satellite signals are blocked or multi-reflected to cause large localization errors.

A reliable large-scale and long-term VPR system has to address several challenges. The first one is to choose a proper image view direction, assuming a non-omnidirectional camera which is commonly available on smartphones unlike panoramic cameras. In a self-driving car scenario, using front-view images from a dash-mounted camera whose view direction is parallel to the driving/street direction is almost a default choice. In fact, most VPR methods have been investigated and evaluated under such front-view conditions, where features of roads, shapes of skylines, and textures of the roadside buildings can all contribute to describing and discriminating various image locations.

Yet not all downstream applications prefer front-view images. For example, a person with low vision might need image-based wearable navigation assistance to find the entrance of a particular shop on the street. The aforementioned front-view images typically contain more than half of the pixels on the road and the sky while the remaining pixels on street sides that are far away from the image capturing locations. Contrarily, the side view offers fronto-parallel images of buildings along streets, which stores enough information required for such an application scenario.

However, currently, there is a lack of datasets that could evaluate VPR methods specifically on side-view images in comparison with front-view ones. Many datasets contain no side-view images or mix them with front-view images without explicit labels (see Table I). Moreover, as far as we know, there is no systematic comparison of VPR performance between images from the two view directions. Thus, the following questions remain unclear: are side-view images more challenging for VPR than front-view ones? And if so, why? And how much is the performance difference?

Another challenge for large-scale VPR systems is data privacy, receiving an increasing attention from the community [12]. Building such systems in large metropolitan areas requires collecting images for a long term, inevitably creating concerns of both violating the privacy of the identity information of individual pedestrians and vehicles, and potentially even leaking their spatial-temporal trajectories. Unlike the privacy-preserving technique in [12] that still operates on raw images, another way could be directly anonymizing the images by wiping out all the identity-related pixels (see Figure 3). However, this again brings some unanswered questions for VPR: would such data anonymization significantly affects the performance of existing VPR algorithms? If so, does it increase or decrease the VPR accuracy/robustness?

This paper aims at filling the gaps by a new VPR dataset and benchmark. This is a year-long dataset captured outdoors by vehicles with front-view and side-view cameras traveling in a metropolitan area recording more than 200,000 GPS-tagged images. This allows us to answer the above questions by comparing the results under various conditions.

The contributions of this paper are as follows:

• •

  We present NYU-VPR, a unique large-scale, year-long, outdoor VPR benchmark dataset containing both front-view and side-view GPS-tagged images taken at different lighting conditions with seasonal and appearance changes in a busy and crowded urban area of New York City. This dataset and our benchmark code will be released for educational and research purposes.

• •

  We benchmark the performance of several popular VPR algorithms with a focus on the influence of image view directions. As far as we know, this is the first work to systematically demonstrate and analyze the causes of the significant challenge of VPR with side-view images.

• •

  We anonymize the identity information in this dataset by removing pixels of both pedestrians and vehicles to address the privacy concerns of large-scale VPR in urban scenes. This is also the first result to show that all the benchmarked VPR algorithms are only marginally affected by this anonymization.

Because NYU-VPR contains only outdoor images used for visual place recognition, we review publicly available datasets that have similar characteristics. The main differences between those datasets and our proposed dataset are summarized in Table I.

Side-view and side-view label: In recent years there has been substantial growth in the number of visual place recognition datasets in the urban areas. However, most datasets only contain front-view images, gathered by cameras on the front and back of cars [3, 4, 7, 8, 9, 11]. The side-view images featuring the storefront and sidewalk are not included. Few datasets include the side-view images in addition to the front-view images. For example, the images in Tokyo 24/7 dataset were gathered by pedestrians’ phones and featured both front-view and side-view images [5]. The images in Pittsburgh dataset were perspective images generated from Google Street View panoramas of the Pittsburgh area [6]. But those datasets do not label the images as front-view or side-view. Thus no work can use those datasets to compare the visual place recognition results on the side-view images versus those on the front-view images. NYU-VPR contains images labeled as front-view and images labeled as side-view. We focus on evaluating the visual place recognition algorithms in both categories. We compare the results of algorithms on the side-view images versus the results on the front-view images in order to analyze the influence of view direction on the long-term visual place recognition.

Dynamic objects, crowded-area and anonymization: Dynamic objects such as pedestrians and vehicles in images may affect the performance of visual place recognition due to the changing appearance at the same place or the existence of obstructing the street view. Besides, the presence of pedestrians and vehicles in publicly available datasets may raise privacy issues if images are not anonymized. There are few appearances of dynamic objects in datasets of images gathered in suburban areas. For example, Nordland is a dataset of images taken on the train on a railway line between the cities of Trondheim and Bodø [3]. In contrast, dynamic objects appears much more frequently in the datasets featuring urban areas [4, 5, 6, 7, 8, 9, 10]. We define crowded areas as metropolitan areas such as New York City and Tokyo that has a high population density and is crowded with pedestrians and vehicles. In Table I, images from Tokyo 24/7, MSLS, and NYU-VPR are gathered in crowded areas. Anonymization is needed on those datasets for privacy protection. Tokyo 24/7 [5] only applied face redaction on pedestrians, and MSLS [10] only anonymize faces and license plates. We use MSeg [13] to replace pedestrians and vehicles with white pixels.

Seasonal changes: Matching images that are taken at the same location in different seasons is crucial for long-term visual place recognition. This is because objects on images change with the seasons: new storefront, trees withering, constructions finished, etc. In Table I, Pittsburgh [6] includes images in different seasons but few image locations are visited in all four seasons. Nordland [3], Oxford RobotCar [9], NCLT [11], and MSLS [10] all include images in four seasons for most locations. NYU-VPR is similar but temporally denser: most locations are visited more than once every month, as shown by the time (Fig. 2(b)) and space (Fig. 4(f) and Fig. 4(e)) distributions of images. So NYU-VPR can be used to evaluate long-term visual place recognition for the influence of seasonal changes.

Baseline methods: Current VPR methods can be roughly grouped into three categories: deep-learning methods, non-deep-learning methods, and methods that only use deep learning descriptors. We select methods in each category. Deep learning methods [14, 15, 16, 17] use a convolutional neural networks (CNN) and train CNN in an end-to-end manner directly. We select NetVLAD [14] and PoseNet [15] in this group. For non-deep-learning methods [18, 19, 20, 5], two classical ones are bag-of-words (BoW) model and Vector of Locally Aggregated Descriptors (VLAD). In this group, we choose DBoW+ORB [18, 21], which was used in the popular ORB-SLAM for loop closing [22], and VLAD+SURF [19, 23], which was used in [24]. Methods that only use deep learning descriptors take advantage of deep nets’ ability to detect a richer set of key points, such as SuperPoint [25] which we also adopted.

Our dataset is named NYU-VPR. It contains images recorded in Manhattan, New York from May 2016 to March 2017. The images were recorded with GPS tags by smartphone cameras installed on the front, back, and side parts of (undisclosed) fleet cars with auto-exposure²²2The raw data was sampled from a larger dataset provided by Carmera.. The dataset contains both side-view images and front-view images. There are 100,500 side-view and 101,290 front-view raw images, each with a $640\times 480$ resolution. On the basis of raw images, we use MSeg [13], a semantic segmentation method, to replace moving objects such as people and cars with white pixels. Fig. 3 compares anonymized and raw images.

The images were recorded on streets around Washington Square Park. The trajectories of the locations where the images were recorded are shown in Fig. 4(d). Since the cameras were placed on fleet cars, and their routes were random, the frequencies of locations where the images were taken are different. The frequencies of the locations where the side-view and front-view images were recorded are shown in Fig. 4(f) and Fig. 4(e) respectively.

Fig. 2(b) shows the time distribution. Since it contains images captured from May 2016 to March 2017, our dataset includes all four seasons. Therefore, it contains various changes of weather, illumination, vegetation, and road construction. As shown in Fig. 2(a), we can see image changes at the same location as the season changes.

Difficulty Level: We assign each side-view query image a difficulty level of easy, medium, or hard. First, we extract SIFT [26] features for each image. Then for each query image, we find the top-8 closest side-view training images by GPS coordinates. The query image and its top-8 closest images form eight image pairs. We use RANSAC to compute a fundamental matrix and the number of inliers for each pair of images. We use three intervals to measure the difficulty level of matching each pair based on each pair’s number of inliers points: 0-19 (hard), 20-80 (medium), $>$80 (easy). The interval values are determined by artificially viewing the image pairs and checking the similarity of the image pairs. The difficulty level of each side-view query image is the most common difficulty level of its eight pairs.

Uniqueness: Our dataset is unique in two ways. First, comparing to front-view images where sky and road surfaces occupy large areas, side-view images focus on street views such as shop signs and metro entrances. Second, we include image anonymization to protect the privacy of pedestrians and cars. In the meantime, anonymized images provide VPR algorithms static and environment-only information, getting rid of moving objects and pedestrians.

Front-view vs. Side-view: We hypothesize that side-view images are more challenging than front-view images for existing VPR methods, based on several theoretical reasons and observations. First, as illustrated in Fig. 5(a), since the images were taken on a moving vehicle at a low frequency, the overlap of two sequential side-view images would only occupy a limited part of the whole image. In contrast, the overlap of two sequential front-view images occupies more space. Furthermore, if the road is narrow and stores are very close to the vehicle, the spatial area covered in the side-view image is less than that covered in the front-view image at the same location. Besides, trees may impact the success rate for side-view images because trees will block features such as banners and entrances. Most trees look the same and thus may cause the uncertainty of localization.

Another important reason is that images are taken on a moving vehicle, suffering from motion blur. When the spatial area coverage of side-view images is less than front-view images, the motion blur effect is more serious on side-view images. We set a threshold for the variance of Laplacian to detect the blurriness of an image³³3https://www.pyimagesearch.com/2015/09/07/blur-detection-with-opencv/. The result in Fig. 5(b) right shows that side-view images have a larger percentage of blurry images than front-view images. We think blurry images are more challenging for VPR and we will analyze the relationship further in the experiment section.

Other Challenges: In addition to the side-view images, there are several other challenges in our dataset. Because our dataset is one-year long, the images taken at the same location have artificial or natural differences. First, Fig. 4(a) left was taken in October 2016 with sideway constructions and the right was taken in December 2016 after the construction. At some locations, the construction may cover the whole image. Second, different seasons cause different appearances at the same location. Fig. 4(b) left was taken in summer, July 2016, and the right was taken in winter, January 2017. In this case, the vegetation in Washington Square Park had changed a lot and snow was covering the ground in winter. Furthermore, if the vehicle was moving fast, the images taken by the vehicle will be blurry (Fig. 4(c)). Although two images were taken at the same location, the blurry one will cause more difficulty during VPR.

After the large-scale experiments and analysis, we can finally answer our questions with confidence. Are side-view images more challenging for VPR than front-view ones? Yes, and the performance drops of all VPR methods are significant, although the dataset has no significant spatial/temporal differences on the distribution of images captured from the two view directions. Would our data anonymization significantly affect the performance of existing VPR algorithms? No, and for some methods, the anonymization could even bring marginal improvements, potentially due to the removal of those VPR noises. Our future work includes benchmarking more VPR methods and with geometric verification.

We would like to thank Carmera for providing the raw NYC image data set that we used for creating the NYU-VPR. This research is funded by the Connected Cities for Smart Mobility towards Accessible and Resilient Transportation (C2SMART), a Tier 1 University Center awarded by U.S. Department of Transportation under contract 69A3351747124.