Directionally Decomposing Structured Light for Projector Calibration

Our calibration device consists of planar pinhole-array masks and a flat-bed scanner. We place the masks to meet the following two requirements: (1) they are located between a projector to be calibrated and the scanner, and (2) they do not occlude each other when viewed from the projector’s point of view (Fig. 2). Because parallel planes do not provide additional constraints for Zhang’s method, the masks are rotated by several degrees from the scanner plane and the other masks. In detailing our method, we assume that the number of the pinhole-array masks is two, which preserves generality.

Zhang’s method accurately calibrates a projector as long as the projected calibration patterns are focused on fiducial planes. Our technique is based on Zhang’s method; however, we use the pinhole-array masks and the scanner surface, rather than a checkerboard, as the fiducial planes. We place the calibration device directly in front of the projector lens. Consequently, projected calibration patterns are strongly blurred on the pinhole-array masks in most cases (Fig. Directionally Decomposing Structured Light for Projector Calibrationa). Nevertheless, our technique can always obtain chief rays by directionally decomposing projected light. Our technique virtually converts a projector of arbitrary aperture sizes and focusing distances to a pinhole projector, by which we can accurately obtain the 2D coordinates of projected pixels corresponding to predefined points on the fiducial planes, such as pinholes, even when projected images are defocused.

Our calibration device directionally decomposes projected light as follows. As a simple case, suppose a single projector pixel $p$ is turned on (white), and the other pixels are turned off (black). Light rays emitted from the white pixel are refracted at different parts of the projector lens (Fig. 3a). These refracted rays are then projected onto a pinhole-array mask $m$, where they do not converge into a point, forming a defocused pixel. When a pinhole $h_{m}$ locates inside the defocused pixel, one of the rays passes through it and hits a point $s(p)$ on the scanner surface. In this manner, our calibration device can extract a ray emitted from a projector pixel in a specific direction.

When we turn on all the pixels, light blobs appear on the scanner surface. Each blob is formed by rays that are emitted from adjacent projector pixels and pass through the same pinhole (Fig. 3b). We denote the blob of a pinhole $h_{m}$ as $b(h_{m})$. The shape of a blob is determined by the shape of the lens aperture and the pose of the scanner relative to the projector. The rays incident on the edge of a blob come from the edge of the projector lens. Because a projector’s lens aperture is generally a circle, the blob shape is an ellipse whose eccentricity becomes zero if the scanner surface is parallel to the principal plane of the lens. A blob $b(h_{m})$ is the intersection of the scanner surface with an oblique cone whose apex and bases are the pinhole $h_{m}$ and the lens, respectively (Fig. 4).

When using Zhang’s method that assumes a pinhole projector, we need to extract a chief ray that passes through the optical center of the lens from all the rays incident on each blob $b(h_{m})$. In other words, we need to identify the projector pixel $p_{c}(h_{m})$ emitting the chief ray. However, doing so is not trivial. The chief ray is identical to the axis of the oblique cone described in Sect. 2.1 and also shown in Fig. 4. Thus, on the scanner surface, the point where the chief ray is incident (i.e., $s(p_{c}(h_{m}))$) is shifted from the center of the blob’s ellipse. Therefore, simply computing the center of the blob does not provide us with the chief ray pixel. The distance between these two points increases when the eccentricity of the blob’s ellipse increases. If the scanner surface is completely parallel to the principal plane of the projector lens, the blob becomes a true circle, and these two points coincide. However, it is difficult to manually realize such geometric alignment because a user does not know where the principal plane exists.

We propose a technique to extract the chief ray of each blob, which works even when the shape of the blob is an ellipse of any eccentricity. Instead of extracting the chief ray on the scanner surface, our technique identifies the projector pixel emitting the chief ray (i.e., $p_{c}(h_{m})$) on the projector’s image plane. In this technique, we assume that the projector’s image plane and the principal plane of the lens are parallel, which is generally achieved in the manufacturing of projectors.

As shown in Fig. 5, rays incident on a blob $b(h_{m})$ can be regarded as emitting from a virtual light source $v(h_{m})$ located behind the projector image plane. We consider an oblique cone whose apex and base are $v(h_{m})$ and the projector lens, respectively. The lateral surface of the cone consists of rays that are incident on and refracted at the edge of the lens. The rays then pass through the pinhole $h_{m}$ and finally become incident on the edge of the blob $b(h_{m})$. According to the above-mentioned parallel assumption, the intersection of the cone with the projector image plane forms a true circle. Therefore, the chief ray is emitted from the projector pixel at the center of the circle. We identify $p_{c}(h_{m})$ by projecting points in the blob on the scanner surface back to the projector’s image plane, fitting a circle to the back-projected blob and computing its center. The back-projection can be easily done once we determine the 2D coordinates of the projector pixels incident on the blob. We identify the 2D coordinates by projecting a series of structured light patterns (e.g., gray code patterns), measuring the resultant illuminance patterns on the scanner surface, and decoding the scanned patterns. Because a back-projected blob forms a circle on the projector image plane regardless of the eccentricity of the ellipse of the blob on the scanner surface, this technique can reliably identify the projector pixel emitting the chief ray in any geometrical relationship between the projector and the scanner. Because back-projection is done simply by looking up the correspondences, we do not require any other information, such as the poses of the pinhole-array masks.

Once the projector pixel of the chief ray $p_{c}(h_{m})$ is identified, we obtain the corresponding scanner surface point $s(p_{c}(h_{m}))$ by inversely looking up the decoded result of the gray code pattern projection. We also identify the corresponding pinhole $h_{m}$ using the following two-step method. First, we divide the blobs on the scanner surface into two groups, each of which corresponds to each mask $m$. Because the masks do not overlap from the projector’s point of view (Sect. 2.1), the blobs can be divided using a simple clustering algorithm such as K-means. Second, we apply a circles-grid recognition algorithm to each group to identify the pinholes. As a result, we obtain a set of corresponding 3D object points on the scanner surface and the mask planes (i.e., $h_{m}$ and $s(p_{c}(h_{m})$) and 2D image points on the projector’s image plane (i.e., $p_{c}(h_{m})$) (Fig. 6). We denote each set of corresponding points as $c(h_{m})=\{h_{m},s(p_{c}(h_{m})),p_{c}(h_{m})\}$. Sets of corresponding points are then directly used in Zhang’s method (see Appendix A) to calculate the intrinsic matrix of the projector.

The accuracy of estimated intrinsic parameters potentially decreases primarily due to image noise. To alleviate this problem, we apply a robust estimation technique and exclude outliers in the calibration parameter estimation. First, we estimate the intrinsic matrix of the projector and its extrinsic matrix relative to the scanner surface using all the sets of corresponding points. The intrinsic and extrinsic matrices are denoted as $\tilde{\textbf{K}}_{0}$ and $[\tilde{\textbf{R}}|\tilde{\textbf{t}}]_{0}$, respectively. We then used these matrices to computationally project each projector pixel $p_{c}(h_{m})$ onto the scanner surface, which we denote as $\tilde{s}_{0}(p_{c}(h_{m}))$. Suppose the reprojection error of each set of corresponding points on the scanner surface is $||s(p_{c}(h_{m}))-\tilde{s}_{0}(p_{c}(h_{m}))||$, we denote its mean over all sets of corresponding points as $e_{0}$. Then, we exclude a set of corresponding points $c_{0}(h_{m})$, which has the largest reprojection error, and estimate the calibration parameters again, which we denote as $\tilde{\textbf{K}}_{1}$ and $[\tilde{\textbf{R}}|\tilde{\textbf{t}}]_{1}$. After $n$ iterations of this process, we estimate the calibration parameters $\tilde{\textbf{K}}_{n}$ and $[\tilde{\textbf{R}}|\tilde{\textbf{t}}]_{n}$ by excluding $n$ sets of corresponding points with the largest reprojection errors (i.e., $c_{0}(h_{m}),\ldots,c_{n-1}(h_{m})$). We expect that the averaged reprojection error $e_{n}$ decreases when $n$ is increased, and, at some point, $e_{n}$ might start to increase because excluded sets of corresponding points include inliers as well as outliers. Therefore, when the averaged reprojection error reaches a local minimum, we stop excluding the outliers and regard the estimated intrinsic and extrinsic matrices at this iteration as the most accurate parameters.

We built a calibration device consisting of two pinhole-array masks and a flat-bed scanner (Canon LiDE220, 216 mm$\times$297 mm, 4,800 dpi) (Fig. Directionally Decomposing Structured Light for Projector Calibrationa). Each mask consisted of a thin black acrylic sheet (thickness: 0.5 mm) and a thick transparent acrylic plate (thickness: 5.0 mm). We poked 19$\times$24 pinholes whose diameters were 0.3 mm into the black acrylic sheet at 5 mm intervals using a laser cutter (Universal Laser Systems, ILS9.75). The pinhole diameter was experimentally determined so the contrast of the scanned image was not significantly degraded by diffraction. The inter-pinhole distance was determined as follows. For accurate calibration, the number of 2D-3D correspondences (i.e., $c(h_{m})$) needed to be increased. This was achieved by increasing the number of pinholes through shortening the inter-pinhole distance. However, a too-short inter-pinhole distance caused an overlap of light blobs on the scanner surface. Considering this trade-off, we experimentally determined the distance of our prototype. The transparent acrylic plate was used to support the pinhole sheet. We made a rectangular hole through the plate using the same laser cutter so that the pinholes were not covered by the transparent acrylic plate, while the black acrylic sheet was rigidly supported when we glued the sheet and the plate together (Fig. 7). We attached a rear-projection screen film onto the scanner surface so that projected patterns could be sufficiently scattered for measurement by the scanner. To validate the versatility of the proposed method, we prepared three projectors with different lens diameters and field of views (FOV): (1) a DLP projector (BenQ MS524, 800$\times$600 pixels, lens diameter: 24 mm, FOV: middle), (2) a second DLP projector (NEC NP110J, 800$\times$600 pixels, lens diameter: 33 mm, FOV: wide), and (3) a 3-LCD projector (Epson EH-TW8400, 1920$\times$1080 pixels, lens diameter: 64 mm, FOV: narrow). In the rest of this paper, we refer to these projectors as (1) DLP1, (2) DLP2, and (3) LCD. Because we could not directly measure the aperture size of each projector, we manually measured the lens diameter as a substitute for the aperture size.

The masks and the scanner were fixed by black aluminum frames and acrylic jigs, as shown in Fig. Directionally Decomposing Structured Light for Projector Calibrationa. All the projectors were placed in the same position. The distance of a projector from the masks and from the scanner were approximately 250 mm and 450 mm, respectively. The masks were rotated 20 degrees around the yaw axis, and the scanner was rotated 45 degrees around the pitch axis. Figure Directionally Decomposing Structured Light for Projector Calibrationa also shows the appearances of the masks and the scanner surface when the LCD projector projected a fringe pattern. The pattern was not visible on the masks due to the defocus, while it was visible on the scanner surface and formed light blobs, each of which corresponded to a pinhole. To confirm the necessity of the pinhole-array masks, we projected the finest fringe pattern (the least significant bit of gray code pattern) and scanned the projected result in two conditions: whether or not the masks were placed between the projector and the scanner. Figure 8 shows the scanned results. When the masks were not placed, the scanned result was almost uniformly white and could not obtain any spatial information of the projected pattern. On the other hand, a fringe pattern is clearly visible in each blob in the scanned result with the masks. Therefore, we confirm that the masks effectively decomposed the structured light.

We validated our method’s calibration accuracy by comparing it with a conventional camera-based calibration approach. Specifically, the three projectors used in this study were calibrated with the proposed and conventional methods under two conditions regarding the distance from the projectors to the plane of sharp focus. We adjusted the focusing ring of each projector so that the projected imagery appeared focused 1 m (the near condition) and 3 m (the far condition) from the projector lens.

We validated the proposed chief ray extraction technique. As described in Sect. 2.2, a blob on the scanner surface forms an ellipse when the scanner surface is not parallel to the principal plane of the projector lens. In this case, the chief ray is not incident on the center of the ellipse. Thus, we extract a projector pixel emitting the chief ray by back-projecting the blob onto the projector’s image plane, on which the blob appears as a true circle whose center corresponds to the chief ray pixel. We compared the calibration results of our method with and without the proposed chief ray extraction technique.

Figure 12a shows a scanned blob when a uniform white image was projected from the DLP1 projector with a focusing distance of 1 m. The blob formed an ellipse whose center is indicated by a green dot in the figure. Figure 12b shows the back-projected blob on the projector’s image plane, which formed a circle rather than an ellipse. The center of the circle is indicated by a red dot in the figure. The red dot is also overlaid on Fig. 12a, which indicates the scanner surface point corresponding to the center of the back-projected blob. As we expected, the green and red dots are not identical in Fig. 12a. The “proposed (naïve)” column of Table 2 presents the calibration result of the DLP1 projector in the near condition when we assume that the chief ray hits the green dot. For comparison, we present the calibration results of the same projector in the near condition from Table 1a. We found that the calibration parameters did not vary significantly among the three methods except $c_{y}$. As discussed in Sect. 3.2, the variance of $c_{y}$ among the methods is common in projector calibration. On the other hand, the MRPE was much larger (over 1 pixel) in the proposed method with the naïve chief ray extraction than the other methods. In general, a calibration result with an MRPE of larger than 1 pixel is a poor result and not recommended for use in an application. Therefore, we confirm that our chief ray extraction technique, based on the back-projection of the scanned blobs onto the projector’s image plane, is essential in providing an accurate calibration result.

As described in Sect. 2.3, our method applies a robust estimation technique in the intrinsic parameter calibration. Figure 13 shows the reprojection errors in the scanner image coordinate system at different numbers of excluded outliers in the calibrations of the DLP2 and LCD projectors in the near condition. The range of the number of outliers was from 0% to 10% of all the sets of corresponding points $c(h_{m})$. We excluded the correspondence in order from large to small reprojection error. As expected in Sect. 2.3, when increasing the number of outliers, the errors initially decreased before beginning to increase. Specifically, the error began to increase after 7.6% and 3.3% of all the sets of corresponding points were excluded in the DLP2 and LCD data, respectively. The same trends were confirmed in the results of the other projectors and the other conditions. Therefore, we confirm that the outlier exclusion positively affected our calibration method. The number of outliers to be excluded should be carefully determined. Specifically, we recommend using the local minimum of the reprojection errors by increasing the number of outliers up to 10% of all the sets of corresponding points.

Figure 14 shows a scanned image when a uniform white image was projected by the LCD projector in the near condition. We overlaid red circles onto the blobs that our method judged to be outliers. In the same way, we overlaid green circles onto inlier blobs and yellow circles onto blobs whose chief ray pixels were not extracted due to missing gray codes. From this result, we confirm that the outliers are distributed at blobs in the periphery of each pinhole-array. The figure also shows that the outlier blobs do not form an ellipse. This possibly occurred because the thickness of our pinhole-array masks is not infinitesimal, and, consequently, the light ray incident on a pinhole from a small grazing angle did not pass through the pinhole. Despite the presence of outliers, intrinsic calibration was as accurate in our method as in the conventional camera-based method, as demonstrated in Sect. 3.2. Therefore, we consider that the outliers do not significantly affect calibration accuracy once they are appropriately excluded.

We conducted a dynamic PM experiment to experimentally validate the usefulness of calibration parameters estimated by our method. The experimental setup is depicted in Fig. Directionally Decomposing Structured Light for Projector Calibrationb. We used the DLP1 projector in this experiment. A 3D-printed Stanford bunny (90$\times$140$\times$130 mm) was used as a projection surface. The bunny was tracked by an off-the-shelf motion capture system consisting of 8 cameras (NaturalPoint, OptiTrack Prime 17W). The projector was placed approximately 3 m from the working volume. The size of the projected image became 2.7$\times$1.5 m at a place where the bunny was moved. We estimated the pose of the projector in the world coordinate system using the intrinsic parameters and four correspondences between the world coordinate and the projector’s image coordinate, which were obtained manually.

Figure Directionally Decomposing Structured Light for Projector Calibrationb shows dynamic PM results. We confirmed that a projection image was rendered in each frame such that a projected texture appeared attached to the surface. This means that both the intrinsic parameters calibrated by our method as well as the pose of the projector estimated using the intrinsic parameters were accurate enough for a dynamic PM application. Therefore, we confirmed that our technique allows a user of a dynamic PM application to calibrate their projectors without placing a large fiducial object at different places with different poses in the large working volume of the application.