Relative Pose Estimation for Stereo ROLLING SHUTTER Cameras

First of all, we will illustrate the differences between the GS model and the RS model in the form of image formation. GS cameras expose all pixels simultaneously, thus the projection can be expressed as: $\lambda_{i}\mathbf{x}_{i}=\boldsymbol{\rm K}(\boldsymbol{\rm R}\mathbf{X}_{i}+\mathbf{T})$, where $\boldsymbol{\rm R}$ and $\mathbf{T}$ represent the rotation and translation of the camera in the world coordinate respectively, $\boldsymbol{\rm K}$ denotes the intrinsic calibration matrix, $\mathbf{X}_{i}$ is the 3D point which is projected to $\mathbf{x}_{i}=[u_{i},v_{i},1]^{T}$ in image plane with a depth value $\lambda_{i}$.

For RS camera, each scanline will correspond to different poses when the camera is moving during image acquisition. We assume that the camera scans in rows. So both $\boldsymbol{\rm R}$ and $\boldsymbol{\rm T}$ can be described as functions of image row $u_{i}$. The projection process under RS camera model is expressed as:

$\displaystyle\lambda_{i}\mathbf{x}_{i}=\lambda_{i}[u_{i},v_{i},1]^{T}=\boldsymbol{\rm K}[\boldsymbol{\rm R}(u_{i})\mathbf{X}_{i}+\mathbf{T}(u_{i})].$

(1)

Next, we derive our linear solver of the stereo RS cameras under consecutive frames.

As illustrated in Fig.1(b), we use the continuous camera motion model in this paper. For clarity, we denote the left and right camera at the first moment and the second moment as ${\rm I}_{1}$, ${\rm I}_{2}$, ${\rm I}_{3}$, ${\rm I}_{4}$ respectively.

We suppose that stereo cameras undergo constant velocity motion among consecutive frames. As shown in Fig.2, we define the total readout time of one frame as $T_{a}$ and the delay time between consecutive frames as $T_{b}$. The stereo cameras are rigidly connected through the baseline, so $T_{a}$ and $T_{b}$ of the left and right cameras are identical. $\{\mathbf{w}_{1},\mathbf{w}_{2},\mathbf{w}_{3},\mathbf{w}_{4}\}\in so(3)$ and $\{\mathbf{d}_{1},\mathbf{d}_{2},\mathbf{d}_{3},\mathbf{d}_{4}\}\in\mathbb{R}^{3}$ represent the rotation and translation velocities of the first row on ${\mathrm{I}_{1}}$, ${\mathrm{I}}_{2}$, ${\mathrm{I}}_{3}$ and ${\mathrm{I}}_{4}$. Furthermore, we assume that the stereo cameras move around the center of the baseline, so we set pose of the center of baseline as ${\mathbf{w}_{0}=\mathbf{0},\mathbf{d}_{0}=\mathbf{0}}$, and the length of baseline is set as $2\mathbf{b}$. Therefore, the pose of first row on ${\mathrm{I}}_{1}$ can be represented as ${\mathbf{w}_{1}=\mathbf{0},\mathbf{d}_{1}=\mathbf{b}}$, and the pose of first row on ${\rm I}_{2}$ is ${\mathbf{w}_{2}=\mathbf{0},\mathbf{d}_{2}=-\mathbf{b}}$.

Suppose that the relative motions between consecutive frames are $\mathbf{w}=[w_{1},w_{2},w_{3}]^{T}$ and $\mathbf{d}=[{\rm d}_{1},{\rm d}_{2},d_{3}]^{T}$, the rotation and translation velocities of each row on each frame can be computed through linear interpolation. The pose of row $u_{i}$ on ${\rm I}_{1}$ is represented as:

$\displaystyle\mathbf{w}(u_{1})=hu_{1}\mathbf{w},\quad\mathbf{d}(u_{1})=hu_{1}\mathbf{d}+\mathbf{b},$

(2)

where $h=\varphi/N$, $\varphi=T_{a}/(T_{a}+T_{b})$ is the readout time ratio, $N$ is the total number of rows per frame. Similar to ${\rm I}_{1}$, the pose of row $u_{3}$ on ${\rm I}_{3}$ is expressed as:

$\displaystyle\mathbf{w}(u_{3})=(1+hu_{3})\mathbf{w},\quad\mathbf{d}(u_{3})=(1+hu_{3})\mathbf{d}+\mathbf{b}.$

(3)

Same as before, the pose of row $u_{2}$ on ${\rm I}_{2}$ and pose of row $u_{4}$ on ${\rm I}_{4}$ are:

$\displaystyle\mathbf{w}(u_{2})=hu_{2}\mathbf{w},\quad\mathbf{d}(u_{2})=hu_{2}\mathbf{d}-\mathbf{b},$

(4)

and

$\displaystyle\mathbf{w}(u_{4})=(1+hu_{4})\mathbf{w},\quad\mathbf{d}(u_{4})=(1+hu_{4})\mathbf{d}-\mathbf{b}.$

(5)

Assume that the relative rotation between consecutive frames is small, the mapping between the rotation matrix $\boldsymbol{\rm R}$ and the rotation vector $\boldsymbol{w}$ can be approximated as: $\boldsymbol{\rm R}\simeq{\rm exp}(\mathbf{w})=\boldsymbol{\rm I}+[\mathbf{w}]_{\times}$. Therefore, we can develop the essential matrix between ${\rm I_{1}}$ and ${\rm I_{3}}$ as:

	$\displaystyle\boldsymbol{\rm E}_{u_{1},u_{3}}=\left[\mathbf{t}_{u_{1},u_{3}}\right]_{\times}\boldsymbol{\rm R}_{u_{1},u_{3}},$		(6a)
	$\displaystyle\boldsymbol{\rm R}_{u_{1},u_{3}}=\boldsymbol{\rm R}_{u_{3}}\boldsymbol{\rm R}_{u_{1}}^{T}=\boldsymbol{\rm I}+(1+hu_{3}-hu_{1})[\mathbf{w}]_{\times},$		(6b)
	$\displaystyle\mathbf{T}_{u_{1},u_{3}}=\mathbf{d}(u_{3})-\boldsymbol{\rm R}_{u_{1},u_{3}}\mathbf{d}(u_{1}).$		(6c)

In this section, we employ the Semi-Global Matching (SGM) algorithm [11] to estimate the depth map from the stereo image pair, and combine the camera pose of each row obtained in the previous section, the correction of the RS images can be completed. First, we use the pose and depth of each pixel in the RS image to back-project each pixel into 3D space, and then reproject 3D points to the image plane corresponding to the pose of the first row as follow:

	$\displaystyle\mathbf{X}_{i}=\boldsymbol{\rm R}_{u_{i}}^{T}[\lambda_{i}{\boldsymbol{\rm K}}^{-1}{[u_{i},v_{i},1]}^{T}-\mathbf{T}_{u_{i}}],$		(13a)
	$\displaystyle{\mathbf{x}_{i}}^{{}^{\prime}}=\boldsymbol{\rm K}[\boldsymbol{\rm R}_{0}\mathbf{X}_{i}+\mathbf{T}_{0}].$		(13b)

We figure out that for stereo RS images, a relatively accurate depth map can be obtained by the SGM algorithm, the depth can be further applied to the correction to get a smoothly undistorted image. Note that the stereo RS cameras enable both 6 DoF camera motion estimation and accurate depth map estimation, which results in our improved RS correction.

To verify the effectiveness of our proposed algorithm, we simulate matching points that satisfy the RS projection model. In the simulation experiments, we set the size of the image as $900\times 900$ with an 810 focal length. Then, we give the value of relative translation $\mathbf{d}$ and relative rotation $\mathbf{w}$. The baseline length $\mathbf{b}$ and the readout time ratio $\varphi$ are also set. When all parameters are known, the correspondences between ${\rm I}_{1}$ and ${\rm I}_{2}$ and correspondences between ${\rm I}_{3}$ and ${\rm I}_{4}$ can be generated. To ensure the creditability, each experimental value is the mean value of 300 repetitions. Moreover, in order to check the robustness and practicability of the algorithm, we do not use RANSAC or nonlinear optimization in the simulation experiments, and experimental results are compared with the experimental results of the GS algorithm [12].

To demonstrate the qualitative results of our relative pose solver, we generate RS images by the simulator and 3D models as in [9]. We first set the ground truth of all parameters such as $\mathbf{d}$, $\mathbf{w}$ and $\varphi$, then we can calculate the pose of each row on each frame in Fig.2. According to the poses, the simulator will generate GS images, then synthesize RS images from these GS images by extracting the corresponding row pixels of the corresponding GS image. In experiments, we set the image resolution as $900\times 900$ with a 1384.6 focal length, the magnitude of $\mathbf{d}$ is 0.3, and the magnitude of $\mathbf{w}$ is fixed to $0.4\degree$. Besides, both translation and rotation include motion on x, y, and z-axis. Then we use SIFT [14] to extract feature points and RANSAC to reduce the effect of noise on pose estimation. Our experimental result is reported in Fig.(4). Fig.4(a) shows the original RS image. And in Fig.4(c).(f), we can observe the depth truth of RS image and the depth error between depth truth and depth estimated by the SGM algorithm. We notice that the left part of the depth error map is set to 0 because the SGM algorithm cannot estimate the depth value of the left area on the left image. Fig.4(b) shows the corrected image using our method and the depth estimated by the SGM algorithm. In Fig.4(d), we overlay the original RS image and the ground truth GS image rendered by the pose of the first row, the red area shows the distortion. And then we overlap the corrected image and the GS image as shown in Fig.4(e). Comparing Fig.4(d) and Fig.4(e), we can find that the rectified RS image almost coincides with the GS image, except for the surrounding area where the SGM algorithm cannot estimate the depth and a small area where the depth error is more than 2m. The experiments prove that our proposed algorithm can also estimate the motion well, and the SGM algorithm can be used to estimate the depth of stereo RS images and get an accurate depth value which can be used in RS image correction.