Accurate calibration of multi-perspective cameras from
a generalization of the hand-eye constraint

Hand–eye calibration problems can be divided into two cases. As shown in Figure 2 and 2, the eye-on-hand case seeks the transformation between a rigidly attached end-effector (i.e. the hand) and camera (i.e. the eye), and the eye-to-base case seeks the transformation between a fixed camera and the base of the robotic arm. The problems are equivalent from an algebraic perspective, and we use the eye-to-base case for our further explanations. We assume that we have an intrinsically calibrated multi-camera system which is fixed in the world coordinate frame. Considering that we have a regular calibration target moving in front of each camera, we define $\mathbf{A}_{i}^{j}$ as the $i$th relative transformation of the fixed camera $j$ to the moving target, and let $\mathbf{B}_{i}^{j}$ be the corresponding transformation of the robot base coordinate frame to the hand coordinate frame, where $j\in\left\{1,\cdots,m\right\}$ and $i\in\left\{1,\cdots,N_{j}\right\}$. $\mathbf{A}_{i}^{j}$ can be easily solved by utilizing PnP methods [30, 31, 32] with a known calibration pattern size, and $\mathbf{B}_{i}^{j}$ is directly obtained from the robot arm system—for which tracking system markers on the target are used lateron. Let $\mathbf{X}^{j}$ furthermore be the transformation from the robot-base coordinate frame to the fixed camera $j$. Finally, let $\mathbf{Y}$ be the transformation from the hand coordinate to the target coordinate frame. Note that $\mathbf{A}_{i}^{j}$, $\mathbf{B}_{i}^{j}$, $\mathbf{X}^{j}$ and $\mathbf{Y}$ are represented by a $3\times 3$ rotation matrix $\mathbf{R}$ and a $3\times 1$ translation vector $\mathbf{t}$.

Note that the following exposition considers only a single camera, so index $j$ is dropped. The standard hand-eye/robot-world calibration constraint is given by

$$\mathbf{A}_{i}\mathbf{X}=\mathbf{Y}\mathbf{B}_{i},$$

(1)

and most solvers solve the problem in two stages (first rotation, then translation):

	$$\displaystyle\mathbf{R}_{\mathbf{A}_{i}}\mathbf{R_{X}}=\mathbf{R_{Y}}\mathbf{R}_{\mathbf{B}_{i}}$$		(2)
	$$\displaystyle\mathbf{R}_{\mathbf{A}_{i}}\mathbf{t_{X}}+\mathbf{t}_{\mathbf{A}_{i}}=\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{i}}+\mathbf{t_{Y}}.$$		(3)

The equations can be solved by either closed-form or iterative solutions. In this paper, we focus on closed-form solutions. As illustrated in [26], we can apply the Kronecker product to represent (2) and (3) as linear equations, thus resulting in

	$$\displaystyle\begin{pmatrix}-\mathbf{I}&\mathbf{R}_{\mathbf{B}_{i}}\otimes\mathbf{R}_{\mathbf{A}_{i}}\end{pmatrix}\begin{pmatrix}vec(\mathbf{R_{Y}})\\ vec(\mathbf{R_{X}})\end{pmatrix}=\mathbf{0}$$		(4)
	$$\displaystyle\begin{pmatrix}\mathbf{I}&-\mathbf{R}_{\mathbf{A}_{i}}\end{pmatrix}\begin{pmatrix}\mathbf{t_{Y}}\\ \mathbf{t_{X}}\end{pmatrix}=\mathbf{t}_{\mathbf{A}_{i}}-\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{i}},$$		(5)

where $vec(\mathbf{R_{Y}})$ and $vec(\mathbf{R_{X}})$ are vectorized rotation matrices. Note that in practice, many such constraints are stacked into larger linear problems. Next, finding the nullspace for the first part of (4) is equivalent to find an efficient and unique solution of

$$\small\begin{pmatrix}n\mathbf{I}&-\sum\limits_{i=1}^{n}\mathbf{R}_{\mathbf{B}_{i}}\otimes\mathbf{R}_{\mathbf{A}_{i}}\\ -\sum\limits_{i=1}^{n}\mathbf{R}^{T}_{\mathbf{B}_{i}}\otimes\mathbf{R}^{T}_{\mathbf{A}_{i}}&n\mathbf{I}\end{pmatrix}\begin{pmatrix}vec(\mathbf{R_{Y}})\\ vec(\mathbf{R_{X}})\end{pmatrix}=\mathbf{0}.$$

(6)

The normal equations of a linear system $\mathcal{A}\mathcal{X}=\mathcal{B}$ can be rephrased as $\mathcal{A}^{T}\mathcal{AX=A}^{T}\mathcal{B}$, which leads to another simplified solution for the second part of (5) as well. The nullspace and thus the rotations $vec(\mathbf{R_{Y}})$ and $vec(\mathbf{R_{X}})$ are easily found by singular value decomposition of (6). The translation is then recovered by substituting $\mathbf{R_{Y}}$ and $\mathbf{R_{X}}$ into the second constraint, and solved for example by applying Cholesky factorization.

Suppose that the $i$th pose of the $j$th camera with respect to the target ($\mathbf{A}_{i}^{j}$) and the corresponding transformation of the robot base frame to the hand frame ($\mathbf{B}_{i}^{j}$) have already been identified. The multi-camera hand-eye/robot-world calibration can be easily formulated as:

$$\mathbf{A}_{i}^{j}\mathbf{X}^{j}=\mathbf{Y}\mathbf{B}_{i}^{j}.$$

(7)

It is important to realize that once multiple such constraints are stacked into a large linear problem, the resulting equation is different from (1). (7) can be used to calculate $\mathbf{Y}$ as a unique variable for calibrating the complete, generalized camera system. Stacking all pose measurements, the rotation constraint becomes

$$\left\{\begin{aligned} \mathbf{R}_{\mathbf{A}_{1}^{1}}\mathbf{R}_{\mathbf{X}^{1}}&=\mathbf{R_{Y}}\mathbf{R}_{\mathbf{B}_{1}^{1}}\\ \mathbf{R}_{\mathbf{A}_{2}^{1}}\mathbf{R}_{\mathbf{X}^{1}}&=\mathbf{R_{Y}}\mathbf{R}_{\mathbf{B}_{2}^{1}}\\ &\cdots\\ \mathbf{R}_{\mathbf{A}_{N_{m}}^{m}}\mathbf{R}_{\mathbf{X}^{m}}&=\mathbf{R_{Y}}\mathbf{R}_{\mathbf{B}_{N_{m}}^{m}}\\ \end{aligned}\right..$$

(8)

Inspired by the definition of (4), all sub-constrains in (8) can now be grouped into the linear problem

$$\begin{pmatrix}\small-\mathbf{I}&\mathbf{R}_{\mathbf{B}_{1}^{1}}\otimes\mathbf{R}_{\mathbf{A}_{1}^{1}}&&\\ -\mathbf{I}&\mathbf{R}_{\mathbf{B}_{2}^{1}}\otimes\mathbf{R}_{\mathbf{A}_{2}^{1}}&&\\ \\ &&\cdots&\\ \\ -\mathbf{I}&&&\mathbf{R}_{\mathbf{B}_{N_{m}}^{m}}\otimes\mathbf{R}_{\mathbf{A}_{N_{m}}^{m}}\\ \end{pmatrix}\begin{pmatrix}[c]vec(\mathbf{R_{Y}}^{\;})\\ vec(\mathbf{R}_{\mathbf{X}^{1}})\\ \\ \cdots\\ \\ vec(\mathbf{R}_{\mathbf{X}^{m}})\end{pmatrix}=\mathbf{0}.$$

(9)

In the spirit of (6), we can again find an efficient and unique solution to the homogeneous linear system (9) by moving to

$$\small\begin{pmatrix}N_{1}\mathbf{I}&-\mathbf{K}_{1}&&\\ \cdots&&\cdots&\\ N_{m}\mathbf{I}&&&-\mathbf{K}_{m}\\ -{\mathbf{L}_{1}}&N_{1}\mathbf{I}&&\\ \cdots&&\cdots&\\ -{\mathbf{L}_{m}}&&&N_{m}\mathbf{I}\\ \end{pmatrix}\begin{pmatrix}[c]vec(\mathbf{R_{Y}}^{\;})\\ vec(\mathbf{R}_{\mathbf{X}^{1}})\\ \cdots\\ vec(\mathbf{R}_{\mathbf{X}^{m}})\end{pmatrix}=\mathcal{U}\mathcal{X}=\mathbf{0},$$

where

$$\mathbf{K}_{j}=\sum\limits_{i=1}^{N_{j}}\mathbf{R}_{\mathbf{B}^{j}_{i}}\otimes\mathbf{R}_{\mathbf{A}^{j}_{i}}\;,\mathbf{L}_{j}=\sum\limits_{i=1}^{N_{j}}\mathbf{R}^{T}_{\mathbf{B}^{j}_{i}}\otimes\mathbf{R}^{T}_{\mathbf{A}^{j}_{i}}.$$

(10)

Note that $N_{j}$ is the number of pose measurements for the $j$th camera. The nullspace of $\mathcal{U}$ can still be efficiently computed by singular value decomposition. The exact solution of $\mathcal{U}\mathcal{X}=0$ is given by the column of the right-hand nullspace matrix $\mathbf{V}$ corresponding to the smallest singular value. In order to recover the rotation matrices $\mathbf{R_{Y}}$ and $\mathbf{R}_{\mathbf{X}^{j}}$, we de-vectorize the solution and obtain the $3\times 3$ matrices $\mathbf{M}_{\mathbf{X}^{j}}=vec^{-1}(\mathbf{R}_{\mathbf{X}^{j}})$ and $\mathbf{M}_{\mathbf{Y}}=vec^{-1}(\mathbf{R_{Y}})$. In order to ensure that $\mathbf{M}_{\mathbf{X}^{j}}$ and $\mathbf{M}_{\mathbf{Y}}$ both satisfy the side-constraints of $SO3$ elements, we conclude with a normalization. We first obtain

$$\left\{\begin{aligned} &\mathbf{R}_{\mathbf{X}^{j}}=\text{sign}(\mathbf{M}_{\mathbf{X}^{j}})\text{det}(\mathbf{M}_{\mathbf{X}^{j}})^{-\frac{1}{3}}\mathbf{M}_{\mathbf{X}^{j}}\\ &\mathbf{R_{Y}^{\quad}}=\text{sign}(\mathbf{M_{Y}})\text{det}(\mathbf{M_{Y}})^{-\frac{1}{3}}\mathbf{M_{Y}}\end{aligned}\right.,$$

(11)

which ensures that we have right-hand matrices of determinant 1. Finally we orthogonalize the matrices by using SVD.

In order to recover the translation $\mathbf{t_{Y}}$ and $\mathbf{t}_{\mathbf{X}^{j}}$, we start by substituting the rotation matrices $\mathbf{R_{Y}}$ and $\mathbf{R}_{\mathbf{X}^{j}}$ from Sec. IV-A into equation (3):

$$\left\{\begin{aligned} \mathbf{R}_{\mathbf{A}_{1}^{1}}\mathbf{t}_{\mathbf{X}^{1}}+\mathbf{t}_{\mathbf{A}_{1}^{1}}&=\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{1}^{1}}+\mathbf{t_{Y}}\\ \mathbf{R}_{\mathbf{A}_{2}^{1}}\mathbf{t}_{\mathbf{X}^{1}}+\mathbf{t}_{\mathbf{A}_{2}^{1}}&=\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{2}^{1}}+\mathbf{t_{Y}}\\ &\cdots\\ \mathbf{R}_{\mathbf{A}_{N_{m}}^{m}}\mathbf{t}_{\mathbf{X}^{m}}+\mathbf{t}_{\mathbf{A}_{N_{m}}^{m}}&=\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{N_{m}}^{m}}+\mathbf{t_{Y}}\\ \end{aligned}\right..$$

(12)

By grouping all sub-constrains in (12), we obtain the linear system:

$$\begin{pmatrix}\small\mathbf{I}&-\mathbf{R}_{\mathbf{A}_{1}^{1}}&&\\ \mathbf{I}&-\mathbf{R}_{\mathbf{A}_{2}^{1}}&&\\ &&\cdots&\\ \mathbf{I}&&&-\mathbf{R}_{\mathbf{A}_{N_{m}}^{m}}\\ \end{pmatrix}\begin{pmatrix}[c]\mathbf{t_{Y}}^{\;}\\ \mathbf{t}_{\mathbf{X}^{1}}\\ \\ \cdots\\ \\ \mathbf{t}_{\mathbf{X}^{m}}\end{pmatrix}=\begin{pmatrix}\mathbf{t}_{\mathbf{A}_{1}^{1}}-\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{1}^{1}}\\ \\ \cdots\\ \\ \mathbf{t}_{\mathbf{A}_{N_{m}}^{m}}-\mathbf{R_{Y}}\mathbf{t}_{\mathbf{B}_{N_{m}}^{m}}\end{pmatrix}.$$

(13)

$\mathbf{t}_{\mathbf{X}^{j}}$ and $\mathbf{t}_{\mathbf{Y}}$ are the solutions to the non-homogeneous linear system $\mathcal{A}\mathcal{X}=\mathcal{B}$, which can again be brought into the normal form $\mathcal{A}^{T}\mathcal{A}\mathcal{X}=\mathcal{A}^{T}\mathcal{B}$ and solved by standard techniques such as Cholesky factorization.

The final eye-to-base transformation for each camera ${\mathbf{X}^{j}}^{-1}$ is given by ${\mathbf{X}^{j}}^{-1}=\begin{bmatrix}{\mathbf{R}_{\mathbf{X}^{j}}}^{T}&-{\mathbf{R}_{\mathbf{X}^{j}}}^{T}\mathbf{t}_{\mathbf{X}^{j}}\\ 0&1\end{bmatrix}$. Extrinsics between cameras in the system can be easily computed by using the chain of transformations

$$\mathbf{T}_{0,j}={\mathbf{X}^{0}}^{-1}\mathbf{X}^{j},$$

(14)

where $\mathbf{T}_{0,j}$ denotes the relative transformation from camera $j$ to camera $0$. We generally use the first camera as the reference frame with respect to which all other extrinsic camera poses are expressed.

The previous subsections have presented a novel closed-form solver for the multi-eye-to-base constraint. A very similar problem is given by the multi-eye-on-hand case, in which we have multiple cameras mounted on the robot arm, and they observe only a single target. In this scenario, we have multiple hand-to-eye transformations $\mathbf{Y}^{j}$, but only a single base-to-target transformation $\mathbf{X}$. The basic constraint of this problem is

$$\mathbf{A}_{i}^{j}\mathbf{X}=\mathbf{Y}^{j}\mathbf{B}_{i}^{j}.$$

(15)

It is interesting to see that this equation is of algebraically identical form, as we may simply take the inverse on either side

$${\mathbf{B}_{i}^{j}}^{-1}{\mathbf{Y}^{j}}^{-1}={\mathbf{X}}^{-1}{\mathbf{A}_{i}^{j}}^{-1},$$

(16)

thus resulting in a form that is entirely similar to (7). From a practical perspective, this means that our generalized solver can be easily used to handle two different cases. The first one is given by the multi-eye-to-base case, which may be more relevant for calibrating larger scale systems that are hard to move. The second is given by the multi-eye-on-hand calibration problem, which may be more relevant to calibrate smaller multi-camera setups. Note that, in the continuation, we will focus on calibrating a vehicle mounted surround-view camera system, hence the remainder of this paper will use the multi-eye-to-base constraint.

Next we introduce the error metrics for comparing our solver against alternatives. Similar to [28], we use the hand-eye constraint (1) to represent rotation and translation errors in the absence of ground truth extrinsics:

$$\small\left\{\begin{aligned} e_{\mathbf{R}}&=\frac{1}{n}\frac{1}{m}\sum\limits_{j=0}^{m-1}\sum\limits_{i=0}^{n-1}angle((\mathbf{R}_{\mathbf{Y}}\mathbf{R}_{\mathbf{B}^{j}_{i}})^{T}(\mathbf{R}_{\mathbf{A}^{j}_{i}}\mathbf{R}_{\mathbf{X}^{j}}))\\ e_{\mathbf{t}}&=\frac{1}{n}\frac{1}{m}\sum\limits_{j=0}^{m-1}\sum\limits_{i=0}^{n-1}\|(\mathbf{R}_{\mathbf{A}^{j}_{i}}\mathbf{t}_{\mathbf{X}^{j}}+\mathbf{t}_{\mathbf{A}^{j}_{i}})-(\mathbf{R}_{\mathbf{Y}}\mathbf{t}_{\mathbf{B}^{j}_{i}}+\mathbf{t}_{\mathbf{Y}})\|\end{aligned}\right.$$

(17)

Please note that we use averaged $\mathbf{R}_{\mathbf{Y}}$ and $\mathbf{t}_{\mathbf{Y}}$ computed from each camera when we are evaluating the solvers designed for the single camera case.

In our simulation experiment, we generate a surround-view camera system that highly resembles the multi-camera system in our real experiments. It has four cameras pointing into all directions (cf. Figure 1). The cameras all lie in the same horizontal plane and have a distance between 0.4 and 0.65m away from the body origin. For the simulated dataset, we add up to 40 camera-to-target poses $\mathbf{B}_{i}^{j}$ for each camera, which are taken from a real sequence and thus have realistic values. We then generate $\mathbf{A}_{i}^{j}$ by using $\mathbf{A}_{i}^{j}=\mathbf{Y}\mathbf{B}_{i}^{j}{\mathbf{X}^{j}}^{-1}$. Note that the iterative method in [28] minimizes reprojection errors in their objective function and therefore requires a simulated camera model and point reprojections. Thus, we only compare our method against other state-of-the-art closed-form solvers in our simulation experiments. We analyze the performance of our method for different noise levels, and varying numbers of cameras and measurements per camera. In practical calibration procedures, the measurements are obtained from well-selected calibration images, thus we do not add any outliers in the simulation experiment. We finally report the accuracy based on above mentioned error metrics.

We compare our method against two alternative closed-form, hand-eye/robot-world methods, denoted (Shah [26] and Li [27]). We conduct three types of experiments:

• •

  Noise level: We use the full 4-camera system and 40 measurements for each camera. Noise is generated by taking a random faction of the absolute coordinates (up to 30%), and adding it directly onto the measurements for both rotation and translation. Note that the camera system is placed in the center of the tracking system’s reference frame, which is why the absolute poses—and thus also the random errors—have a relatively homogeneous distribution across the tracking system’s area. As shown in Figure 4(a), for both Shah and Li, the noise addition leads to a significant increase of the errors, especially for Li. Our proposed method performs best in terms of both rotational and translational errors.

• •

  Number of cameras: We fix the noise ratio to 5% and vary the number of cameras to be calibrated from 1 to 4. The results are illustrated in Figure 4(b). As expected, our method is well-suited for calibrating multi-camera systems. Adding more cameras will not decrease accuracy, while for Shah and Li, the errors increase with the number of cameras in the system.

• •

  Number of measurements: We keep using the full 4-camera system and fix the noise ratio to 5%. We vary the number of measurements for each camera from 3 to 40. Note that 3 is the minimum number of pose measurements for a single hand-eye/robot-world calibration solver. The results are indicated in Figure 4(c). As can be observed, using more pose measurements leads to a large reduction of errors for all methods, and our method maintains a higher level of accuracy than the alternatives, which also proves that well-distributed pose measurements can significantly improve the calibration accuracy.

To conclude, we compare the computational efficiency of different methods. All methods process 160 measurements for 4 cameras in total. Our method’s processing time is 5.54ms, Shah uses 3.58ms, Li uses 14.77ms. As mentioned in [28], the linear Li method will decompose an $8n\times 16$ matrix using SVD, where $n$ is the number of measurements, thus it is 3 times slower than our method and Shah. Our method has comparable efficiency to the fastest alternative.

In order to demonstrate the performance of our algorithm on real platforms, we apply it to two multi-camera systems given by a surround-view camera system and a stereo setup with overlapping fields-of-view (cf. illustrated in Figure 5 and 5). Note that the surround-view camera system is mounted on a mobile rig and first calibrated inside the lab. Only after the calibration is finished, the entire frame is installed on top of the vehicle. The stereo setup allows us to compare our calibration results against a classical stereo calibration method , denoted (GT [8]).

Table I shows our results on both surround-view and stereo camera systems and compares them against all alternatives. The retrieved extrinsic parameters from all methods are listed in Table II. The following is worth noting:

• •

  We carefully select 80 calibration images for the surround-view camera setup and 50 for the stereo setup. We add the iterative methods introduced by [28] as alternatives and select the two best methods based on geometric constraints and reprojection errors respectively, which are c1-Euler-separable and c2-Euler-separable in [28]. As shown in Table I, all algorithms successfully complete the calibration without gross errors in rotations, while the translational errors are much more obvious due to error propagation. This is especially true for Li and Tabb methods. Similar to our simulation experiments, our method again outperforms all alternatives including iterative methods.

• •

  A direct comparison of the retrieved extrinsic parameters is shown in the Table II. As can be observed, our calibration result with non-overlapping assumption is nearly as accurate as classical algorithms exploiting overlap in the fields-of-view.