Quasi-calibration method for structured light system with auxiliary camera

The FPP adopts the concept of phase to find corresponding pixels between the camera and the projector. Using phase instead of intensity has many advantages in reconstructing 3-D geometry; the reconstructed 3-D result becomes more noise tolerant because the system designer can manage the light sources. Moreover, it is less affected by surface reflection and color of the texture. To find the phase value via the system, a set of fringe pattern should be projected onto the target and captured by the camera. The projected pattern should include at least three frames of phase-shifted fringe patterns. This methodology, which utilizes multiple frames to recover phase values, is called the N-step phase shifting method. It finds the phase value $\phi$ by solving the equations as follows,

where $I^{\prime}(u,v)$ is the average intensity and $I^{\prime\prime}(u,v)$ is the intensity modulation of corresponding pixel $(u,v)$ at the $k-$th frame. For example, if there are $N$ frames of phase-shifted images, phase $\phi$ can be found by solving Eq. (1) for $\phi$ with the least square method. Then the solution $\phi(u,v)$ is represented as follows,

However, due to the characteristic of the inverse tangent function, the retrieved phase ranges from $-\pi$ to $\pi$ with $2\pi$ discontinuities. The discontinuities result in having multiple points with the same phase value, meaning that it cannot set one-to-one correspondence between the camera and projector pixel. A procedure called phase unwrapping is required to resolve the issue, making the retrieved phase continuous and absolute. The phase values are unwrapped with the equation as follows,

where $\Phi(u,v)$ represents the unwrapped phase, the phase without discontinuities, and $K(u,v)$ represents the proper integer added to the phase value to connect the discontinuities. Here, we have denoted the unwrapped phase as $\Phi(u,v)$ to distinguish it from the phase $\phi(u,v)$, which has discontinuities in each period. Temporal phase unwrapping, used in the following experiments, connects the discontinuities by projecting additional binary patterns containing the order $K(u,v)$ of the fringe period.

Every optical device is represented as a pinhole model. The pinhole model is a mathematical expression of the relationship between the 3-D world coordinates $(x,y,z)$ and the image pixel $(u,v)$. The linear pinhole model is represented mathematically as follows,

where $s$ is a scaling factor and A,R and t are

respectively. A represents the intrinsic parameters of the imaging device, which includes the effective focal length $f_{u}$ and $f_{v}$, principal point $c_{u}$, and $c_{v}$ and skew factor $\gamma$. For the research level camera, $\gamma=0$. If we define projection matrix P as

linear pinhole model can be turned into,

Then, the linear pinhole model of the camera and projector can be expressed as follows,

where the superscripts c and p represent the camera and projector respectively, and the superscript T represents the transpose of a matrix. Fig. 1 shows the common configuration of the FPP system. If the FPP system is calibrated, the projection matrix $\textbf{P}^{\text{c}}$ and $\textbf{P}^{\text{p}}$ is determined. Then, the FPP system finds the world coordinates by solving Eq. (8) and Eq. (9). If the camera pixel $u=u_{0}$ and $v=v_{0}$ is taken into account when solving the equation, the corresponding world coordinates are described as follows,

where $p^{\text{c}}$ and $p^{\text{p}}$ represent elements of the projection matrix of the camera and projector. If the fringes are projected, phase values will be generated with the N-step phase-shifting method. When the vertical fringes are projected, the phase values on the camera pixel can define the corresponding projector pixel as follows,

where $T$ represents the fringe periods in pixels. If we replace $u_{\text{p}}$ with Eq. (11), and solve Eq. (10) with respect to $(x_{0},y_{0},z_{0})$, the world coordinates can be expressed as follows,

which show that the world coordinates$(x_{0},y_{0},z_{0})$ are a function of the absolute phase value $\Phi$ because the other values are constant.

Previous studies in fringe projection profilometry have shown that the phase value $\Phi$ and the world coordinates $(x,y,z)$ have a certain relationship. According to Zhang[22], the relationship can be approximated by the third-order polynomials for each pixel. The polynomial model defines the relationship between the phase value $\Phi$ and world coordinates $(x_{\text{w}},y_{\text{w}},z_{\text{w}})$ in each camera pixel as follows

where $a_{0},...,a_{3},b_{0},...,b_{3},c_{0},...,c_{3}$, are the coefficients for the polynomial model, in which each pixel has its own unique value. A fringe projection profilometry system calibrated with a pixel-wise polynomial model has been shown to reconstruct the 3-D geometry with higher accuracy, especially on the edges of the image. However, the polynomial model performs well only in the short depth range. To overcome that limitation, Vargas et al. calibrated the fringe projection profilometry system using another pixel-wise calibration model called the rational model[25]. The relationship between $\Phi$ and $(x^{\text{c}},y^{\text{c}},z^{\text{c}})$ in the rational model is as follows,

where $m_{0},m_{1},m_{2},m_{3},m_{4}$, are the calibration coefficients for each pixel. While a polynomial model has been assumed based on the experimental data without backbone equations, the rational model was derived from the pin-hole model equation. Compared with the polynomial model, the rational model has achieved a larger depth range of the reconstruction area while maintaining the high accuracy of the pixel-wise calibration method.

Calibration of a projector in a FPP system is more complicated than calibration of a camera because the projector is output device. Despite previous efforts, such as Zhang’s projector calibration method utilizing camera-captured images[18], calibrating the projector still remains a sophisticated process[26, 27, 28, 29, 30]. In this context, we have developed a system calibration method for FPP system that does not require a projector calibration process throughout the sequence. Moreover, the preceding FPP systems were constrained to employing expensive Digital Light Processing (DLP) projectors as illumination devices. DLP projectors are required because two-directional fringe projection is required for calibration, even though the reconstruction requires only one-directional fringe pattern. However, the proposed method overcomes this limitation by enhancing the system’s adaptability depending on the equipment conditions. Calibration of the projector-camera system will be possible, only if the projection can generate a one-directional phase map with repeatability. This method uses one projector and two cameras. Since we can remove the second camera after the calibration process, we named the second camera an auxiliary camera, and the first one, a main camera. Fig. 2 compares the system configuration during the calibration and reconstruction. The proposed method calibrates the system by finding the relationship between a phase value and world coordinates in each pixel of the main camera. Here, to attain world coordinates in each pixel, the auxiliary camera is temporarily used during the calibration. If the system is calibrated, 3-D geometry can be measured without it. The main camera and projector acquire depth information using previously derived phase-coordinate relations. We named this novel method quasi-calibration method. The proposed calibration process consists of the following steps.

• 1.

  Step 1: Calibrate the main and auxiliary camera with the standard stereo vision calibration method to obtain the intrinsic and extrinsic parameters.

• 2.

  Step 2: Find the 3-D geometry of the calibration board with stereo cameras based on a conventional stereo-matching algorithm with the phase information calculated by the N-step phase shifting algorithm.

• 3.

  Step 3: Determine the ideal planar model that fits the three-dimensional geometry of the calibration board by utilizing the plane fitting equation.

• 4.

  Step 4: Find the relationship between the 3-D geometry, $(x^{\text{c}},y^{\text{c}},z^{\text{c}})$, which is centered by the main camera and $\Phi$ for each pixel in the pixel-wise calibration model.

• 5.

  Step 5: Using only the main camera and digital projector, capture the fringe images of the object(s) to be measured and reconstruct the 3-D geometry based on the previously established relationship.

While finding the 3-D geometry of the calibration board (Step 2), single-directional fringe patterns are projected, and an absolute phase map is generated to give guidance on finding the corresponding pixels of each camera. However, the generated absolute phase map contains phase errors along the boundaries of the calibration circles, resulting from the high contrast between the white circle and a black background. To remove certain artifacts, the phase map was fitted on a smooth polynomial surface[22]. Additionally, before fitting the observation data to the pixel-wise calibration model, the closest flat plane was generated via the equation of the plane as following

where the variables A, B, C, and D are determined to minimize the offset error between the observation and the ideal plane (Step 3). The coefficients of the pixel-wise calibration model are retrieved with the point cloud values of an ideal plane and absolute phase values (Step 4). Using an ideal plane instead of an initial observation clears out the inherent distortions of the optical devices. If coefficients of the pixel-wise calibration model are found, the system is calibrated at this moment, and the auxiliary camera can be removed if necessary. The 3-D shape of the target can be measured only with the projector and the main camera. The following experiment presents 3-D results of the FPP system calibrated with the proposed method. It is proved that this method calibrates the FPP system without projector calibration and successfully reconstructs a complex object. Furthermore, the following experiments conducted with a plastic slit illuminator show that the proposed method allows users to utilize wider types of fringe illumination devices.

To validate whether the proposed calibration method can also calibrate the system with a rough type illuminator, the digital light processing (DLP) projector in the previous experiment setup was replaced by a LED with a plastic slit. Fig. 7 shows the experimental setup of the system and Fig. 8 shows the plastic slits that were used in the experiment. When calibrating the system using the proposed method, the patterns were projected by placing the plastic slit in front of the headlamp. To project phase-shifted patterns, the slits were moved one by one and captured with cameras. The gray code patterns were also projected to unwrap the phase map using the plastic slits, which have been cut according to the shape of the gray code pattern. Fig. 9 describes how the slits were shifted for pattern generations. To remove the phase discontinuities caused by machining error of the slits, the phase values of misaligned pixels were replaced with the median filtered phase values of the nearby pixels. In this experiment, 10 poses of the calibration board were taken with the plastic slit illumination system to calibrate the system. Fig. 10 shows the pattern-projected images of the calibration board and corresponding phase map. After the plastic slit illumination system is calibrated with the proposed method, the system can reconstruct a geometry with a simple configuration as shown in Fig. 11. In order to evaluate the performance of the plastic slit illumination system calibrated with the proposed method, a mirror, sphere, and complex statue were taken with the system. Fig. 12(a) and Fig. 12(b) show the reconstruction result of a flat plane and a sphere repectively. Fig. 12(c) and Fig. 12(d) show the corresponding error map, which presents the offset from the measurement to the ideal geometry. The RMS error of the reconstruction result of the flat plane was 0.1647 mm. The measured radius of the sphere was 50.06 mm which has an error of 0.12 % from the ground truth radius of 50 mm. The RMS error of a sphere reconstruction was 0.2521 mm. Fig. 12(e) and Fig. 12(f) show the corresponding histogram of each error map. Fig. 13 presents the result of reconstructing the complex statue with the plastic slit illumination system.