Automatic Sign Reading and Localization for Semantic Mapping with an Office Robot

Simultaneous Localization And Mapping (SLAM) techniques provide navigational maps that enable robots to autonomously explore an environment. The emerging family of Semantic Mapping techniques add semantic information about locations and objects in the environment to this basic map data. Semantic mapping enables robots to navigate based on a location’s semantics rather than simple geometric poses, and by extension allows the robot to reason about locations using automated planning or based on semantic parses of natural language commands [1]. Semantic maps can be particularly useful for consumer-facing robotic applications, such as aiding visitors in navigating an unfamiliar building [2] or object retrieval [3]. Manually labeling map semantics can be tedious or impractical for large maps. This work focuses on the autonomous semantic labeling of a mapped environment.

Buildings often feature signage that annotates important semantics for the benefit of human occupants, such as placards indicating office numbers. Modern Optical Character Recognition (OCR) methods can be used to extract semantic meaning from text found in optical frames. Prior work used hand crafted detectors to identify door placards, register their position on a map, and transcribe their labels [4] [5]. These methods require either the construction of a map prior to scanning for placards, or for a human to guide the robot while scanning for placards. The contribution of this work is to construct the map and semantics in post-processing of video RGB-D data recorded during exploration.

This work utilizes RGB-D sensor data collected from a robot equipped with an Azure Kinect camera [6]. Video data was recorded during human-guided exploration of an office floor. From the video recording, a navigation map is built using a version of ORB-SLAM [7] which has been modified to improve performance in the presence of very challenging input, in the form of flat-colored walls with few obvious features to match during visual reconstruction (an important step in the ORB-SLAM pipeline). After the navigation map is built, extracted keyframes are utilized to build a full 3D reconstruction of the hallways from sensor data. At the same time, a trained YOLOv5 [8] object-detector is used to identify door placards in each frame. The detector used in this study is specifically trained to detect door number placards. If a placard is detected, it is localized in frame and added to the reconstruction. EAST [9] and Tesseract-OCR [10] are used to read and record each placard’s text. The text contains information such as room numbers, which then is used to mark up the semantics of room locations on the constructed map.

This paper demonstrates an effective semantic mapping system for recorded RGB-D data of an office hallway which reads and localizes room numbers (Sec. III). The success of this work is measured in terms of the number of placards identified, the accuracy of the localization, the ability to make accurate maps in the presence of featureless walls, and the number of placards successfully read.

Simultaneous Localization and Mapping (SLAM) refers to the problem of exploring an environment, mapping the features of that environment, and placing the robot on the map [11]. Semantic Mapping is the problem of placing semantic labels on map features [12] for efficient task completion or natural communication with humans [13][14]. Semantic SLAM tackles both problems at once, often using semantic labels in the localization process [15].

Much of the work on Semantic Mapping and Semantic SLAM, has focused on identifying object labels with the sensor being considered, often with RGB or RGB-D cameras [16][12][17], although identifying rooms and spaces is also a common problem [18]. Objects are placed onto a map using a representative model ranging from detailed 3D models [19], to bounding ellipsoids and cubes [17], or a labelled point cloud and voxel grid [20].

Labels are generally categorized a-priori by type, without discrimination between instances of the same type [21]. An alternative is to discover labels through text in the environment that uniquely identify objects or spaces, thereby improving the specificity of the semantic map. Prior work identified text as a useful indicator of planar objects rich in distinct visual features useful for optical SLAM methods [22][23]. Transcribing text has been used for localization on a given map [24][25]. Text recognition as a part of a SLAM system has also been used in self-driving cars and retail spaces [26][27][28]. A major difference between street signs and office signs is that the former is designed to be easily read from forward facing fast moving vehicles, while the latter by slow moving pedestrians capable of reorienting their view at will. This difference causes many of the challenges discussed in this paper.

There are two major investigations which we expand upon. The earliest work is Autonomous Sign Reading (ASR) for Semantic Mapping [5], and the other is Pose Registration for Integrated Semantic Mapping (PRISM) [4]. While prior work focuses on design of placard detectors, this paper focuses on efficient discovery.

This semantic mapping system constructs maps from video and depth data (RGB-D video) recorded using a Microsoft Azure Kinect [6] mounted on a mobile robot (a University of Texas at Austin Building-Wide Intelligence BWIBot [29]).

Recordings are processed in two phases. In the first phase, a SLAM map is constructed with a modified ORB-SLAM [7]. The second phase extracts semantics from each keyframe. The second phrase comprises several steps:

1. 1.

   Detect placards (Regions of Interest - ROIs)

2. 2.

   Extract planar features

3. 3.

   Identify text lines

4. 4.

   Transcribe text to string

5. 5.

   Validate string

6. 6.

   Map Reconstruction and Projection

7. 7.

   Data aggregation

As each placard is observed, the pose and label is recorded. If the label could not be validated, it is left blank.

A video was recorded on an Azure Kinect sensor [6]. The color camera was set to a resolution of $3840\times 2160$px @ $15$ fps, and the depth camera was set to a binned wide field of view (WFOV) with $512\times 512$px @ $15$ fps. The color camera resolution is set such that placards viewed at far distances and poor angles have the best possible chance of being read. The depth camera is set so WFOV has the best possible overlap with the color camera and the maximum possible depth range for WFOV ($0.25-2.88$m).

Four ORB-SLAM maps were created to evaluate performance. Trial 3 serves as the baseline map. Figure 4 shows the map. The keyframe trajectory is shown in red. The localized placards are shown as blue dots with lines indicating their orientation. Orange markings show placards that have been discarded because they were are not localized close to a wall.

Figure 5 shows the clustering of $\theta_{i}$ values, the green vertical line indicates the selected baseline value for valid values of $\theta^{*}_{i}$, the red vertical lines indicate the predicted clusters given only $\frac{\pi}{2}$ radian angles of all the walls. As shown, the predictions align well with the observations.

Table I summarizes the numerical results. The symbols $\mu,\sigma$ here refer to the mean and standard deviation respectively of the indicated measurements. The displacement error on Trial 3 has been omitted since every placard excepting the duplicates are utilized for the baseline comparison.

There are two placards which every trial missed, one which was the only placard not to conform to the standard measurements, and one which was posted on the window of a dark room. Since the placards are black and the dark room gave the appearance of a black wall, both of these situations presented a unique challenge to YOLOv5 which it was not trained to handle.

Trial 4 missed one additional placard which had been discarded for not being localized close to a wall. Trial 4 and Trial 1 also falsely identify an extra placard on a section of wall with a visual similarity to the preferred objects.

Each trial contains a number of duplicate placards. Trial 2 is the worst offender because it has the worst performing map, containing segments of matching walls that are out of alignment. Placards on these walls are duplicated. Other sources of duplication come from cases where the edge of the detected placard is visually aligned with a protruding corner of a nearby wall. The resulting depth cloud includes points lying on this wall at a much closer depth thereby skewing the placard’s localization.

Figure 7 shows how the displacement errors are distributed for each trial. As the placards get further away from the origin frame, defects in the map cumulatively contribute to the localization error. Therefore, the error tends to be small near the origin, and larger further away. The worst performing trial by this metric is Trial 1, while Trial 4 is the best.

The average error in orientation remained under $6$ degrees of deviation for all trials. Again, Trial 1 has the worst results and Trial 4 the best, closely mirroring the displacement results.

Figure 6 shows the impact of applying point cloud merging on an ORB-SLAM map. ORB-SLAM loses tracking in the corridor on the left-hand side of Figure 3. In this area the robot does a full rotation with blank walls on either side. Without the correction, ORB-SLAM stops tracking as it pans against each wall. Between each wall, a second map is created. After completing the rotation ORB-SLAM recognizes a return to the starting viewpoint, and begins tracking the first map again. As the robot drives back down the corridor, the localization drifts and a falsely mapped second corridor is created. When the robot returns to the half-circle viewpoint, ORB-SLAM merges the first and second maps with an added phantom hallway. However, with the modification described in Sec. III-A a single map is maintained, therefore when the robot reaches the half circle viewpoint mark ORB-SLAM instead does a loop closure which corrects the localization and merges the corridors.

Out of the observed placards, Trial 1 correctly read the labels on $6/35$ ($17\%$), Trial 2 on $14/43$ ($33\%$), Trial 3 on $13/34$ ($38\%$), and Trial 4 on $9/33$ ($27\%$).

A system for adding semantic markers to a SLAM map utilizing naturally placed text markers was presented. The system can be run on recordings of RGB-D data. The system demonstrates a high degree of consistency between mapping results as well as a robustness to featureless regions in the color video. The system is able to identify all but two text markers and read about $30\%$ of those it identifies on average.

This work has taken place in the Learning Agents Research Group (LARG) at the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (CPS-1739964, IIS-1724157, FAIN-2019844), the Office of Naval Research (N00014-18-2243), Army Research Office (W911NF-19-2-0333), DARPA, General Motors, Bosch, Cisco Systems, and Good Systems, a research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone.