Autonomous Removal of Perspective Distortion of Elevator Button
Images based on Corner Detection

Autonomous elevator operation is a promising solution for mobile navigation in office buildings. The system consists of three parts: button recognition, motion planning, and robot control. Among them, button recognition is the most basic but challenging part. Its performance directly determines the success rate and robustness of the entire elevator autonomous operating system. Traditional button recognition algorithms tend to place markers on the elevator button panel in advance, and then the position of each button relative to the markers can be acquired by calculating the geometric relationship between them. Unfortunately, these hand-engineered algorithms are inconvenient and will fail if the elevator button panel cannot be marked beforehand. To overcome this limitation, in the past few years, researchers have proposed plenty of deep learning-based button recognition algorithms [1, 2, 3], which can output button recognition results directly from raw elevator button images. However, the recognition accuracy of these deep learning-based algorithms is not satisfactory due to various image conditions and distortions. There exist many kinds of button shapes, button sizes, elevator panel designs, and light conditions. Meanwhile, various perspective distortions and unexpected blurs make it more challenging to recognize buttons accurately. In this article, we propose a novel deep learning-based approach to autonomously correct perspective distortions of elevator button images based on button corner detection results.

The proposed approach consists of two parts. The first part is a button corner detection algorithm. We train an image segmentation model to perform feature extraction of raw elevator button images and obtain button segmentation results. Then four lines of every button are identified by using the Hough Transform method [4]. Pixel coordinates and the order of all button corners can be obtained as they are the intersections of identified lines. The second part is a pose estimation algorithm. It takes the hypothetical button corners with standard pixel coordinates as reference points and calculates the camera motions to align corners of raw elevator button images and the reference points. Then by using an inverse transformation, new elevator button images without perspective distortions can be generated.

The contributions of this work are summarized as follows:

• •

  We derive detection results of button corners by utilizing an image segmentation model and the Hough Transform method.

• •

  We propose a novel algorithm that can autonomously remove perspective distortions of elevator button images based on the detection results of button corners.

The remainder of this article is organized as follows. The previous work on elevator button recognition and existing distortion removal methods are reviewed in Sec. II. Sec. III and Sec. IV outline the proposed autonomous perspective distortion removal approach, while the experimental results are presented and discussed in Sec. V. Finally, we draw some conclusions and discuss the future work in Sec. VI.

To begin with, we first define the notations which will be frequently used in this paper. Throughout this work, matrices are written as boldface uppercase letters, and vectors are written as boldface lower letters. The following notations are used:

• •

  ${\textbf{C = [}}{{\textbf{c}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{c}}_{N}}{\rm{]}}\in{R^{2\times N}}$ - the detected button corners in the image plane;

• •

  $\mathop{{\textbf{C }}}\limits^{\wedge}{\rm{=[}}{\mathop{\textbf{c}}\limits^{\wedge}_{1}}{\rm{,\cdot\cdot\cdot,}}{\mathop{\textbf{c}}\limits^{\wedge}_{N}}{\rm{]}}\in{R^{3\times N}}$ - the detected button corners in the normalized image plane;

• •

  ${\textbf{U = [}}{{\textbf{u}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{u}}_{N}}{\rm{]}}\in{R^{2\times N}}$ - the presupposed standard button corners without distortion in the image plane;

• •

  $\mathop{\textbf{U}}\limits^{\wedge}{\rm{=[}}{\mathop{\textbf{u}}\limits^{\wedge}_{1}}{\rm{,\cdot\cdot\cdot,}}{\mathop{\textbf{u}}\limits^{\wedge}_{N}}{\rm{]}}\in{R^{3\times N}}$ - the presupposed standard button corners without distortion in the normalized image plane;

• •

  ${\textbf{G = [}}{{\textbf{g}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{g}}_{N}}{\rm{]}}\in{R^{2\times N}}$ - the rectified button corners in the image plane;

• •

  $\mathop{{\textbf{G }}}\limits^{\wedge}{\rm{=[}}{\mathop{\textbf{g}}\limits^{\wedge}_{1}}{\rm{,\cdot\cdot\cdot,}}{\mathop{\textbf{g}}\limits^{\wedge}_{N}}{\rm{]}}\in{R^{3\times N}}$ - the rectified button corners in the normalized image plane;

• •

  ${{\rm{M}}_{{\mathop{\rm int}}}}\,=\,\left[\begin{array}[]{l}{\rm{F/}}{{\rm{s}}_{x}}\quad{\rm{0}}\quad\,{{\rm{o}}_{x}}\\ \;{\rm{0}}\quad\,\,{\rm{F/}}{{\rm{s}}_{y}}\;\,{{\rm{o}}_{y}}\\ \;{\rm{0}}\quad\,\;\;\;{\rm{0}}\quad\,{\rm{1}}\end{array}\right]$ - the intrinsic parameter of the camera;

• •

  ${\rm{F}}$ - focal length in the meter for fixed focal length, non-zoomed camera;

• •

  ${{\rm{o}}_{x}}{\rm{,}}\,{{\rm{o}}_{y}}$ - image center in pixel;

• •

  ${{\rm{s}}_{x}}{\rm{,}}\,\,{{\rm{s}}_{y}}$ - pixel width and height in meter;

• •

  ${\textbf{D = [}}{{\textbf{d}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{d}}_{N}}{\rm{]}}\in{R^{3\times N}}$ - the spatial coordinates of detected button corners;

• •

  ${\textbf{E = [}}{{\textbf{e}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{e}}_{N}}{\rm{]}}\in{R^{3\times N}}$ - the spatial coordinates of standard button corners;

• •

  ${\textbf{M = [}}{{\textbf{m}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{m}}_{N}}{\rm{]}}\in{R^{3\times N}}$ - new spatial coordinates of detected button corners after rotation operation;

• •

  ${\textbf{P = [}}{{\textbf{p}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{p}}_{N}}{\rm{]}}\in{R^{3\times N}}$ - new spatial coordinates of detected button corners with depth equal to 1 after rotation and translation operation;

• •

  ${\textbf{R}}\,({\rm{\theta}})$ - the matrix representation of angle-axis parameterized rotation ${\rm{\theta}}$;

• •

  T - the matrix representation of translation, between detected button corners and standard button corners;

• •

  ${\rm{b}}$ - the number of buttons on image;

• •

  ${{\textbf{K}}_{H}}{\rm{=[}}{{\textbf{k}}_{h1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{k}}_{hN}}{\rm{]}}\in{R^{1\times N}}$ - slopes of the horizontal line of every button in space coordinate;

• •

  ${{\textbf{K}}_{V}}{\rm{=[}}{{\textbf{k}}_{v1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{k}}_{vN}}{\rm{]}}\in{R^{1\times N}}$ - slopes of the vertical line of every button in space coordinate;

• •

  ${\textbf{Cos = [Co}}{{\textbf{s}}_{1}}{\rm{,\cdot\cdot\cdot,}}{{\textbf{Cos}}_{N}}{\rm{]}}\in{R^{1\times N}}$ - cosine values of the angles between horizontal and vertical lines of every button in space coordinate.

The detail of the proposed perspective distortion removal algorithm is shown in Alg. 1.

The first step is to establish a presupposed elevator button image, in which pixel coordinates of the button corners U are standard without perspective distortion. Two types of the presupposed elevator button images are shown in Fig. 4.

The second step is back projection. $\mathop{\textbf{C}}\limits^{\wedge}$ and $\mathop{\textbf{U}}\limits^{\wedge}\in{R^{{\rm{3}}\times N}}$ are obtained by adding a third row $\left[{1\cdot\cdot\cdot\;1}\right]$ to C and $\textbf{U}\in{R^{2\times N}}$. Then the inverse matrix of the intrinsic camera parameter is leveraged to obtain the spatial coordinates of button corners, and the equation is shown as follows:

In this algorithm, we assume that for the standard button corners without perspective distortions, the slopes of horizontal lines equal to zero, the slopes of vertical lines equal to infinity, and the cosine values of the angles between horizontal and vertical lines equal to zero. Thus for E, we have:

The third step is to compute the rotation and translation matrix to form new spatial coordinates of detected button corners. Three-dimensional rotation matrices are utilized to rotate spatial coordinates of the corners, which include rotating ${{\rm{\theta}}_{x}}$ against the x-axis, ${{\rm{\theta}}_{y}}$ against the y-axis and ${{\rm{\theta}}_{z}}$ against the z-axis, shown as follows respectively:

where ${{\rm{\theta}}_{x}}$, ${{\rm{\theta}}_{y}}$, ${{\rm{\theta}}_{z}}$ are radian values and the relation between angle value and radian value is:

The rotation matrix is formed as ${\rm{\textbf{R}(\theta}})=\;{{\textbf{R}}_{x}}\,{\rm{*}}\,{{\textbf{R}}_{y}}\,{\rm{*}}\;{{\textbf{R}}_{z}}.$ Then, new spatial coordinates of detected button corners are computed through the following equations:

where P[3] represents the third row of new spatial coordinates, and the translation matrix T is defined as the difference value between spatial coordinates of the first corner in the presupposed elevator button image and the distorted elevator button image.

The fourth step is to estimate camera motions. The goal is to find the optimal rotation matrix and translation matrix so that the lines formed by the new spatial coordinates of the distorted button corners are parallel to the lines formed by the spatial coordinates of the presupposed standard button corners. In this algorithm, we set $\alpha=-40,\beta=40,\gamma=0.5$, which means that every 0.5 degree against each axis is sampled to form the rotation matrix ${\textbf{R}}({\rm{\theta}})$. The range is from -40 degrees to 40 degrees. Three criteria are chosen to evaluate which formed rotation matrix is optimal:

The first criterion is ${{\textbf{K}}_{H}}$, representing the slopes of horizontal lines of every button in space coordinate. ${{\textbf{k}}_{hi}}$ of each button is defined as:

where ${x_{i}}$ denotes the first value of the $i$-th corner and ${y_{i}}$ denotes the second value of the $i$-th corner, respectively. Then we can obtain the two-norm result of ${{\textbf{K}}_{H}}$,

The second criterion is ${{\textbf{K}}_{V}}$, representing the slopes of vertical lines of every button in space coordinate. ${{\textbf{k}}_{vi}}$ of each button is defined as:

Then we can obtain the two-norm result of ${{\rm{K}}_{V}}$ and its reciprocal ${{\textbf{K}}_{rV}}$,

The third criterion is Cos, representing the cosine values of the angles between horizontal and vertical lines of every button in space coordinate. The horizontal line vector, vertical line vector, and ${\textbf{co}}{{\textbf{s}}_{i}}$ of each button are shown as follows:

where $({x_{1}},{y_{1}},{z_{1}})$ denotes the spatial coordinate of the first corner, $({x_{2}},{y_{2}},{z_{2}})$ the spatial coordinate of the second corner, and $({x_{4}},{y_{4}},{z_{4}})$ the spatial coordinate of the fourth corner. Then we can obtain the two-norm result of Cos,

When ${\left\|{{\textbf{Cos}}}\right\|_{2}}$ is smaller, we can obtain a better perpendicular result of horizontal and vertical lines of buttons.

To combine the three criteria, they are normalized as follows:

Then the final criterion is shown as follows:

When $FinalCR$ is smallest, we can obtain the optimal rotation matrix and translation matrix.

The fifth step is to form new rectified images. After obtaining the optimal pose (${\textbf{R}}({\rm{\theta}})^{{}^{\prime}},{\textbf{T}}^{{}^{\prime}}$), each pixel of the distorted elevator button image can be transformed to have new spatial coordinates through Eq. (7). Then intrinsic camera parameter is used to do projection and get new pixel coordinates in the normalized plane,

By taking the first and second rows, we can obtain pixel coordinates of the rectified button corners in the image plane. Finally, a new rectified elevator button image is generated by applying an inverse image warping operation.

To verify the effectiveness of the proposed approach, we collect a dataset with 15 images from 3 different elevators, which are captured from different angles of views containing severe perspective distortions. The intrinsic camera parameter is:

The value of Eq. (13) is used to measure the accuracy of the proposed perspective distortion removal algorithm, which represents the two-norm value of cosine values of the angles between horizontal and vertical lines of all buttons in space coordinate. When the value of Eq. (13) is smaller, the rectification performance is better. The experimental results of 15 elevator button images are shown in Table I. Some demonstrations of the corresponding original and rectified images are presented in Fig. 5. From Fig. 5 and Table I, we can see that our proposed approach is capable of removing perspective distortions of elevator button images autonomously with high accuracy.

This article proposes a novel deep learning-based approach that can autonomously remove perspective distortions of elevator button images. We utilize an image segmentation model and Hough Transform method to obtain the detection results of button corners. A novel algorithm is designed to correct the perspective distortions of original elevator button images. Currently, the presented algorithm can only handle elevator button images that contain several rectangle buttons. For elevator button images that contain circular buttons, as a circle has no slopes, the presented algorithm will fail. In the next step, we will use the Hough transform method to calculate the center coordinates and radius of the circular buttons and develop a novel algorithm that can autonomously remove perspective distortions of images containing circular elevator buttons.