Indoor 3D Reconstruction with an Unknown Camera-Projector Pair

A camera is represented with a pin-hole model as

The camera matrix is decomposed into a concatenation of camera intrinsics ${\mathbf{K}}$ and extrinsics, i.e. camera rotation ${\mathbf{R}}$ and translation $\boldsymbol{t}$, as

Similar to [26], we assume a “natural” camera with square pixels, i.e. $\gamma$ equals to 1 and $f_{\rm x}=f_{\rm y}=f$. Additionally, the lens distortion is assumed insignificant or corrected in advance, which is thus not considered in the following discussion.

As shown in Fig. 1(c), a typical C2 is a Tri-rectangular tetrahedron with four vertices, e.g. $O,A,B,C$. Each two of vertices defines an edge and hence a total of six edges. There is a special vertex, i.e. $O$ in Fig. 1(c), usually called the right angle (RA), where the angle between each two edges with RA as their common vertex is a right angle. Accordingly, we have three mutually orthogonal edges for a C2, as shown in Fig. 1(c), which are called legs $\boldsymbol{n}_{\rm A}$, $\boldsymbol{n}_{\rm B}$ and $\boldsymbol{n}_{\rm C}$, respectively. The planes defined by each two of these legs are also orthogonal to each other, which are denoted as the faces $\rm{\Pi}_{\rm A},\rm{\Pi}_{\rm B}$ and $\rm{\Pi}_{\rm C}$ with normalized $\boldsymbol{n}_{\rm A}$, $\boldsymbol{n}_{\rm B}$ and $\boldsymbol{n}_{\rm C}$ as their normal respectively, see triangles $\triangle ABO$ and $\triangle ACO$ in Fig. 1(c). Notably, significant occlusion of a C2 is allowed, as shown in Fig. 1(b), where all or some of the vertices and legs are not directly observed. In fact, we require only three partially observed faces of a C2 for calibration, since vertices and legs can be inferred from inter-plane homographies (see Sec. 4.2).

According to the definition above, a C2 is defined by seven parameters up to an unknown scale: three for rotation, two for scaled translation and two for the length ratios between legs. Specifically, in the camera coordinate frame, these parameters can be specified as

and

The core of direct 3D reconstruction with an unknown CPP lies in the CPP self-calibration. Provided a known camera principal point, the CPP self-calibration aims to recover the remaining CPP intrinsics from only two-view correspondences of an unknown C2.

More specifically, given matched image point pairs $\{\boldsymbol{x}_{\mathrm{c,S}}(i),\boldsymbol{x}_{\mathrm{p,S}}(i)\mid i{=}1,2,\\ \ldots,N_{\mathrm{S}}\}$ on faces $\rm{\Pi}_{\rm S}$ of a C2, $S=A,B$ or $C$, respectively, and the camera principal point $\boldsymbol{x}_{\mathrm{c,0}}$, we aim to recover both focal lengths $f_{\rm c}$ and $f_{\rm p}$ of the camera and projector, and the principal point $\boldsymbol{x}_{\mathrm{p,0}}$ of the projector. Extrinsics and the parameters of C2 are last considered in this paper, since they can be easily determined after intrinsics estimated.

Please note that, even $\boldsymbol{x}_{\mathrm{c,0}}$ is assumed known, according to a simple argument counting, the self-calibration is still significantly ill-posed: There are four unknowns, i.e. $f_{\rm c}$, $f_{\rm p}$ and $\boldsymbol{x}_{\mathrm{p,0}}$, whereas the Kruppa equation can only provides two constraints, leaving two unknowns unconstrainted. For textureless indoor scenes with insufficient line segments, we will show how to solve this problem by constructing sufficient constraints from an unknown C2 in the following section.

For clarity, we assume a canonical C2 as shown in Fig. 1(b). Suppose the four vertices $O$, $A$, $B$, $C$ of this C2 are imaged by a camera at image points $\boldsymbol{x}_{\mathrm{O}}$, $\boldsymbol{x}_{\mathrm{A}}$, $\boldsymbol{x}_{\mathrm{B}}$ and $\boldsymbol{x}_{\mathrm{C}}$, respectively. $\boldsymbol{X}_{\mathrm{O}}$, $\boldsymbol{X}_{\mathrm{A}}$, $\boldsymbol{X}_{\mathrm{B}}$ and $\boldsymbol{X}_{\mathrm{C}}$ are inhomogeneous 3D coordinates of these vertices, respectively, in the camera coordinate frame. The camera intrinsic matrix is $\mathbf{K}$. Please note that since only a single view is considered in this subsection, the subscript “c” is omitted.

Each vertex can be obtained by back-projecting from its image as

Since the C2 is parameterized up to an unknown scalar, without loss of generalization, we set $\lambda_{\mathrm{O}}$ to a specific value, say $\lambda_{\mathrm{O}}=1$. Accordingly, only $\lambda_{\mathrm{A}}$, $\lambda_{\mathrm{B}}$ and $\lambda_{\mathrm{C}}$ are unknown.

By enforcing the orthogonality constraints between the legs $\boldsymbol{n}_{\rm A}$, $\boldsymbol{n}_{\rm B}$ and $\boldsymbol{n}_{\rm C}$, we have the following equations.

Substitution Eq. 7 into Eq. 8, we have the simplified form as

Eq. 9 establishes a concise relationship between the IAC and the image of C2, which provides three independent constraints on the camera intrinsics. Moreover, Proposition 1 can be easily derived as below.

Proposition 1. Given a calibrated camera, a C2 can be determined up to at most two different solutions from its image.

Proof. As an alternative parameterization to that in Sec. 2.2, a C2 can also be defined by its four vertices. Given the images $\boldsymbol{x}_{\mathrm{O}}$, $\boldsymbol{x}_{\mathrm{A}}$, $\boldsymbol{x}_{\mathrm{B}}$ and $\boldsymbol{x}_{\mathrm{C}}$ of these vertices, the up-to-scale determination of this C2 is to determine a triplet of three scalars $\lambda_{\mathrm{A}}$, $\lambda_{\mathrm{B}}$ and $\lambda_{\mathrm{C}}$. Similarly, $\lambda_{\mathrm{O}}$ is set to 1. Enforcing the orthogonality constraints yields Eq. 9. Since the camera intrinsics ${\mathbf{K}}$ is known for a calibrated camera, Eq. 9 changes to a ternary quadratic form. Solving this form of Eq. 9 leads to two solutions for the unknown triplet.

There are three possible configurations for the solutions: two real different solutions, one real solution and two complex solutions, as shown in Fig. 2. Here we only provide a geometric interpretation about these configurations instead of rigorous proof (since it is in fact straightforward). The last two configurations correspond to a C2 located with (at least) one vertex (not the RA) close to or behind the camera center, see vertex $B$ close to the camera center $Q$ in Fig. 2(a), which are not possible for a real camera. Hence, we focus only on the configuration with two different real solutions, as shown in Fig. 2(b). These two solutions just correspond to two C2’s with the same common RA but different orientations. As shown in Fig. 2(b), they are “concave” and “convex” C2’s, respectively. Given another view or some prior about the C2, the “convexity” can be easily identified, see convexity check in Sec. 4.2, and the only solution of C2 can thus be determined.

Based on Proposition 1, the two-view geometry can be easily derived. It defines a transfer from camera intrinsics to those of the projector, as shown in Fig. 3.

Camera to C2. Other than vertex images, matches $\{\boldsymbol{x}_{\mathrm{c,S}}(i),\boldsymbol{x}_{\mathrm{p,S}}(i)\mid i{=}1,2,$ $\ldots,N_{\mathrm{S}}\}$ on faces ${\rm{\Pi}}_{\mathrm{S}}$ of the C2 are also considered. $S$ can be $A$, $B$ or $C$, and $N_{\mathrm{S}}$ is the point number of face ${\rm{\Pi}}_{\mathrm{S}}$. Given the camera’s $\mathbf{K}_{\mathrm{c}}$, we first determine the unknown C2 from its vertex images according to Proposition 1. Those vertices $\boldsymbol{X}_{\mathrm{O}}$, $\boldsymbol{X}_{\mathrm{A}}$, $\boldsymbol{X}_{\mathrm{B}}$ and $\boldsymbol{X}_{\mathrm{C}}$ are thus computed, which are represented in the camera coordinate frame. Note that in this specific derivation the world coordinate frame coincides with the camera coordinate frame.

Solving Eq. 10 yields the unique $\lambda_{\mathrm{Ci}}$ and hence $\boldsymbol{X}_{\mathrm{C}}(i)$. Similarly, all points $\{\boldsymbol{X}_{\mathrm{S}}(i)\mid i{=}1,2,\ldots,N_{\mathrm{S}}\}$ on each face can be determined from their images.

$\boldsymbol{X}_{\mathrm{S}}(i)$ to projector. Now, these non-coplanar points $\boldsymbol{X}_{\mathrm{S}}(i)$ and their images $\boldsymbol{x}_{\mathrm{p,S}}(i)$ in the projector’s view are available. The parameter matrix $\mathbf{M}_{\mathrm{p}}$ for the projector can easily be solved via the Direct Linear Transformation (DLT) algorithm [17]. The intrinsics $\mathbf{K}_{\mathrm{p}}$ (as well as $\mathbf{R}$ and $\boldsymbol{t}$ between the camera and projector) can obtained by decomposing $\mathbf{M}_{\mathrm{p}}$.

This process establishes a transferring chain from $\mathbf{K}_{\mathrm{c}}$ to $\mathbf{K}_{\mathrm{p}}$, which can be represented by an abstract function $g(x)$ as

Additionally, if a known principal point $\boldsymbol{x}_{\rm{c,0}}=(x_{\rm{c,0}},y_{\rm{c,0}})$ of the camera is given, Eq. 11 further changes to a univariable function.

According to Eq. 11 and Eq. 12, given a C2, $\mathbf{K}_{\mathrm{p}}$ depends only on $\mathbf{K}_{\mathrm{c}}$ or $f_{\rm c}$. Accordingly, the original multi-variable (i.e. four) CPP calibration problem is simplified to a univariable estimation problem with the only unknown $f_{\rm c}$. Additionally, this relationship between $f_{\rm c}$ and $\mathbf{K}_{\mathrm{2}}$ is deterministic, since given a guess of $f_{\rm c}$, the corresponding $\mathbf{K}_{\mathrm{p}}$ can be computed uniquely without solving ambiguous polynomial equations such as Kruppa equation. This determinacy together with univariability leads to a simple yet reliable calibration algorithm.

According to Eq. 12, we only need to determine the single unknown $f_{\rm c}$ for the CPP calibration. Please note the transferring in Eq. 11 from the camera to projector is invertible. Considering that the forward and backward transferring should be consistent, the camera transferring to the projector should transfer to the same camera if transferring back. This allows us to define a cycle loss between the original and transferred back camera intrinsics. Additionally, since the projector is also assumed “natural”, of which $\gamma_{\mathrm{p}}$ equals to 0 and $f_{\rm px}$=$f_{\rm py}$, we construct an optimization objective as

where

Solving Eq. 13 is a simple univariable optimization problem. Additionally, if a relatively loose feasible range for $f_{\rm c}$, say $f_{\rm c}\in[0,f_{\mathrm{max}}]$, where $f_{\rm max}$ can be set to 10000 or larger, a globally optimal solution can be estimated via an exact search [11]. This is trivial when we discretizing the feasible range with some sampling rate $\Delta f$, say $\Delta f=1$, without sacrificing much accuracy. Alternatively, the solution can be searched via a bounded one-dimensional optimizer[11], which was used in our experiments due to its fast convergence and comparable stability.

Input preparation. Our algorithm requires only matches between the camera and projector views for a C2. We firstly established correspondences using structured-light patterns [28]. These correspondences are then partitioned into three sets on faces of the C2, as shown in Fig. 3(c), by manually drawing respective masks, see details in Supplementary material.

Inference of vertices and legs for an occluded C2. Given only partially observed faces of a C2, the inter-view homography $\mathbf{H}_{\mathrm{S}}$, ${S}$= ${A}$, ${B}$ or ${C}$, induced by each face can be estimated. The leg, say $\boldsymbol{n}_{\rm A}$, between two faces, i.e. $\rm{\Pi}_{\rm B}$ and $\rm{\Pi}_{\rm C}$, can be determined using the two eigen vectors of $\mathbf{H}_{\mathrm{B}}*\mathbf{H}_{\mathrm{C}}^{-1}$[38]. Accordingly, the $\rm RA$ can be determined from the three legs. The other three vertices can be picked manually in the legs.

Convexity check. In our algorithm, the convexity of a C2 is visually recognized by human or known as a prior. For instance, when reconstruction a room corner, as shown in Fig. 1(a), the C2 can be easily recognized as a concave one.

The pseudocode of our algorithm is shown in Fig. 4.

Scenes. Since there is no publicly available datasets of structured-light indoor images, we captured our own data using a CPP (see CCP setup below) to evaluate methods on diverse indoor scenes. As shown in the first row in Fig. 5, each scene contains a partially observed C2, which consists of floors and walls with repetitive or weak textures. Additionally, only limited line segments are observable in the scenes. These makes challenging scenes for CPP self-calibration and reconstruction. To be noted, in spite of different-level occlusions, vertex images of the C2 can still be inferred from its partially observed faces, see the second row in Fig. 5.

CPP setup. A CPP with a 2448×2048 camera and a 854×480 projector was used, which contained constant but different intrinsics across scenes. The ground truths were obtained with a target-based method [35], where the focal lengths and principal points are {1791.1, (1256.3, 1054.3)} and {1247.3, (377.1, 234.0)} for the camera and projector, respectively. More comparison results of different CPPs can be found in Supplementary material.

Baselines. We compared our method with state-of-the-art methods in both two-view and multi-view configurations (see Tab. 1), respectively. As baselines of traditional and learning-based self-calibration, COLMAP [29] and the recently reported DroidCalib [16] were used.

In the multi-view configuration, both methods estimated only camera intrinsics from about 20 views, whereas the projector was not involved since it cannot capture images. This is a typical multi-view SfM task in well-posed configuration.

In the two-view configuration, similarly, DroidCalib still only perform camera calibration but from two views, of which one is identical to the camera view used in our method and the other is close to the projector view. In contrast, COLMAP with different initial value and optimization strategies were performed for a comprehensive comparison, see Tab. 1. For a fair comparison, all of them accept matches from our method as the input, i.e. $\{\boldsymbol{x}_{\mathrm{c,S}}(i),\boldsymbol{x}_{\mathrm{p,S}}(i)\mid i{=}1,2,\ldots,N_{\mathrm{S}}\}$, and then directly estimate the intrinsics of the camera and projector, respectively. The camera principal point (PP) is known a prior. In Tab. 1, “C” refers to imposing the projector image center as its initial values of PP, whereas “R” means a normally random PP from the range $[1,W]\times[1,H]$. $W$ and $H$ are width and height of the projector image, respectively. COLMAP with “BA” performs additional bundle adjustment (BA) [17] upon the result of those without “BA”. Additionally, PP’s were refined in methods with “BA”, which, however, were fixed in those without “BA”.

In addition to methods above, PlaneFormer [3], a state-of-the-art two-view 3D reconstruction method, was also compared for 3D reconstruction evaluation.

Instead of limited in indoor scenes with a CPP, our method can be easily adapted to more general scenarios with only cameras, where we try to self-calibrate cameras in a two-view SfM problem. A constant but unknown camera across views was assumed. With simple modification on the objective in Eq. 13, see details in Supplementary material, the ill-posed problem is solved, as shown in the first row in Fig. 8. Furthermore, since additional constraints can be derived from the assumption of a constant camera, no prior about the camera principal point is required. This allows to estimate all camera intrinsics even from a pair of cropped images, as shown in the second row in Fig. 8. Surprisingly, our method achieved consistent accuracy on cropped images as the original ones, which demonstrates considerable stability and robustness to image cropping. It provides potential solution for reliable camera self-calibration in two-view SfM.

We further investigated the impact of different configurations of the optimization objective, to determine which one is more significant for the calibration. Ten configurations were evaluated, of which the first seven corresponded to $E_{\rm i},i=1,2,…,7$, and the last three were $E_{\rm pro}$, $E_{\rm cycle}$ and $E$ in Eq. 13 and Eq. 14, respectively. As shown in Fig. 9 and Tab. 2, the last two configurations contribute dominantly to the calibration, which achieved the lowest errors across scenes. This indicates that the cycle loss is much more significant than $E_{\rm pro}$. The sum of $E_{\rm pro}$ and $E_{\rm cycle}$, i.e. $E$, achieved comprehensively best performance, reaching the top calibration accuracy.

We also investigated the impact of the number of matches. The original matches were downsampled by different rates, where the average number for the four scenes were reduced from an order of magnitude 9×$10^{4}$ to that of only 100. As shown in Tab. 3, our method performs consistently well with different downsampling rates, demonstrating robustness to the number of matches.

There are still some limitations for the proposed method. On the one hand, our method fails in some degenerated configuration, for instance, when at least one of the faces of a C2 passes through the camera center. This is just the case where all points on a face are imaged to the same line. On the other hand, our method requires accurate matches of three segmented faces of a C2. To achieve this, we manually segment images and use structured light patterns to establish reliable and accurate correspondence across views. In the future work, we will further develop algorithms for automatically detecting faces of a C2 and their matches, to fully automate the calibration and reconstruction.