Face Authentication from Grayscale Coded Light Field

We suggest the following simple model of grayscale LF acquisition. For simplicity, we assume the coding of only two views, $view_{0}$ and $view_{1}$, with two effective coding matrices, $\phi_{0}$ and $\phi_{1}$, drawn as different spatial projections of the coding mask $\Phi$. Each pixel (at the spatial location $u,v$) in the coded image, $CI$, can, thus, be modeled as

$\displaystyle CI(u,v)=\sum_{i}view_{i}(u,v)\phi_{i}(u,v)$

$\displaystyle i=0,1.$

(1)

Full reconstruction of the views can be obtained using sparse coding or deep learning methods. However, for the task of face authentication, we show that no such reconstruction is needed. The coding mask $\Phi$ may have a random distribution, but it can be measured at system calibration and is therefore known. Using this knowledge, we can obtain a fast, sparse reconstruction of the views by exploiting the mask distribution. For a random binary mask, some sensor pixels code only one view. These pixels’ locations can be easily computed by

$$SM_{i}=\mathbb{I}[\phi_{i}>0]\odot\mathbb{I}[\phi_{1-i}=0],$$

(2)

where $SM_{i}$ denotes a ”sparse mask” indicating the pixels in which only $view_{i}$ is captured on the sensor, and $\mathbb{I}$ is the indicator function. Therefore, we may obtain a ”free” reconstructed sparse view by

$$view_{i,s}=CI\odot SM_{i}.$$

(3)

This setup trades resolution for either light efficiency or the number of views being coded. These factors can be optimized to fit a specific application (i.e., more angles can be coded to create a more accurate depth map). Moreover, the coding mask pattern can be optimized given a specific system geometry, which determines the shift between coding matrices. We focus on the case of only two views. It is possible to extend to more using the same technique. The only constraint is the resolution of the pixels that correspond just to one angle. These can be controlled by changing the the resolution of the captured image and the light efficiency. In our case of two angles, we assume light efficiency of $50\%$, which implies as we describe later that in the coded image quarter of the pixels will belong solely to just one of the angles. We exploit the fact that face recognition can be done on low-resolution images, so the loss of information in the encoding process might not fail the verification task.

Our technique is not limited to grayscale images and can be applied also to color images with the appropriate mask or Bayer pattern. Yet, these adjustments complicate the implementation and increase manufacturing cost. Moreover, capturing color information sacrifices resolution and light efficiency, and thus, we focus on the grayscale case showing that depth information can compensate on the absence of color in the face verification task.

Note that $50\%$ light efficiency means that about a quarter of the pixels in each view can be trivially reconstructed. Assuming a 1.3 MP sensor, of $1080\times 1400$ resolution, our reconstructed sparse views yield $540\times 700$ pixels (randomly spaced in the original resolution). Notice that the state-of-the-art RGB face recognition networks receive faces of resolution $250\times 250$, which may indicate that our “free” reconstructions are sufficient for the task of authentication. Our experiments hereafter show that this is indeed the case.

A reliable face authentication system must ensure that an actual face is being captured, rather than a 2D printed image of a face. In order to do so, other verification systems use expensive 3D scanners such as Time of Flight [7] or Structured Light [8] to detect the depth. Besides their cost, notice that the 3D reconstruction process is time consuming. Moreover, some of these systems are power inefficient, sensitive to ambient daylight and only produce a depth signal. LF images, therefore, hold more promise for the task of face authentication, since they capture both depth and intensity information.

We suggest the following simple idea, for detecting whether a coded LF image is of a 3D or a 2D object, without the need to fully reconstruct the object. Our method relies on the fact, that the disparity map of a plane being captured in a stereo setting, is also a plane. Let a plane in 3D space be defined by the equation

$$c=ax+by+z.$$

(4)

In a standard stereo setting, the transformation between Euclidean and image spaces is given by

$$x=\frac{B}{d}(u-u_{0}),y=\frac{B}{d}(v-v_{0}),z=\frac{B}{d}f_{u},$$

(5)

where $B$ is the baseline, $d$ is the disparity measured at the pixel $(u,v)$, $(u_{0},v_{0})$ is the principle point of the image, and $f_{u}$ is the pixel focal length. Combining equations (4) and (5),

$$d=a\frac{B}{c}u+b\frac{B}{c}v+\frac{B}{c}(f_{u}-au_{0}-bv_{0}).$$

(6)

Notice that this implies that in the 2D case, the disparity is affine with respect to the image coordinates. Thus, in the case of a flat, 2D object, one can estimate the entire disparity map from a few sampled points.

This leads us to the following simple anti-spoofing technique. Assume that the captured face is flat. After applying the ”free” reconstruction described in the previous section, a basic interpolation can be conducted, followed by a local disparity calculation at three specified image points. The left sparse view can be projected to a new estimation of the right sparse view, by applying the affine disparity map corresponding to the three calculated disparity values, $D_{plane}$:

$$view^{\prime}_{r,s}(u,v)=view_{l,s}(u+D_{plane}(u,v),v).$$

(7)

A similarity measure can then be applied on the projected right view $view^{\prime}_{r,s}$, and the captured sparse right view, $view_{r,s}$. Naturally, we expect this similarity to be lower for captured 3D objects, which have non-flat disparity maps. We found that comparing average $\ell_{1}$ distance between cubic interpolated sparse images provides good results.

As in many other works, we approach the verification task using metric learning. This way, given a fine, accurate metric, we can determine whether two faces belong to the same person or not. We show that combining a multi-view and a multi-modal (having both depth and intensity information) approach, with the power of a large scale pre-trained neural network, enables competitive results in the face verification task on grayscale coded LF data.

Inspired by MVCNN [9], which deals with 3D objects shape retrieval rather than faces, we applied a multi-view approach, having different CNN branches learning from 2D projections of a 3D face. Other than the original work, we handle multi-modal data so the weights are not shared and the different input features are concatenated instead of pooled.

The network structure may be summarized as follows. Each input, i.e., left view, right view, and depth map, is fed into an independent Inception Resnet V1 [10] backbone. We choose this network since it has a public pre-trained model available [11], trained on the 3.3 million faces of VGGFace2 [12]. We then concatenate all the output features and feed them to two fully connected layers. We used triplet loss [1] on the final features to learn the embedding.

To test our anti-spoofing concept, we conduct the following experiment on synthetic data. Using the GD LFW database, which we have created and resembles the depth resolution achieved by our system, we project each grayscale left view to a ”flat” right view, by randomized disparity plane parameters. Then, we simulate the acquisition process, resulting in sparse views of both the real and the ”flat” projections. Given a sparse left view and a sampled disparity, we create the projected sparse right view, as described in Section 2.2. We measure the similarity between the captured (simulated) and projected sparse right views, by the $\ell_{1}$ loss on their bicubic interpolations. The similarity distributions are plotted in Fig. 2a, showing a clear separation of the flat and deep faces.

We repeated the process on the GLF Faces dataset, captured by our system partial prototype. It contains 35 2D views of flat and curved surfaces, such as partial cylinders, and 459 pairs of views of real faces. We randomly sampled 35 3D views to preserve the scale of the histograms, shown in Fig. 2b. Though it does not exhibit a clear cut in the separation as in the synthetic case, also in this case, our method can still save a lot of heavy computation time on obvious spoofing cases, which may be the most common type. Notably, the fit error of curved surfaces is still distinctively smaller than that of actual faces, even though they are non-planar. The ROC curves of our anti-spoofing $\ell_{1}$-error based classifier are shown in Fig. 2c. Having the verification done also on depth images afterwards prevents the more complicated spoofing scenarios. Yet, it will be applied only on a small fraction of the 2D scans, which were not detected in the first “cheap” anti-spoofing test.

As we are the first to work with coded LF data, we compare our method to the landmarks depth (LDF) method in [19] which is the-state-of-the-art on the full LF database presented in [20]. This is the only relevant method for comparison in terms of the data being processed and the computation resources. We report the results using the same terms and measures as in [20] on our coded LF dataset.

We repeated Experiment 2 in [19], training for binary classification (genuine or spoofing), mixing all attack types (warped and planar paper, in our case). While keeping the amount of examples of each class equal, we split the data with a $25\%$ test set. We report the average classification error rate (ACER) at the equal error rate (EER), for 50 experiments with random data split. Evaluating both strategies on GLF Faces, we outperform their result with ACER of 0.039, compared to 0.08 achieved by LDF. On the synthetic GD LFW, we are also better with $0.004$ ACER compared to $0.05$ achieved by LDF. The results are summarized in Table 1.

We report results following the unrestricted, labeled outside data protocol of LFW [21], with 10-fold cross-validation. We report a competitive 99.5% average accuracy on the GD LFW data. For comparison, the original pre-trained network achieves 99.6% accuracy on LFW, and 81.1% on grayscale LFW. Fine tuning the pre-trained model (i.e., on a single ”branch” of Inception ResNet V1) on a three channel image containing two grayscale views and a depth channel, we got 90% accuracy, which shows the benefits of our architecture.

On Grayscale LF Faces, the network trained on the synthetic LFW data achieves 91.2% accuracy on randomly sampled pairs of matching and mismatching identities. End-to-end fine-tuning on our system data, increases the accuracy to 98.75%. The test is done in an open set manner, having 78 people in the training set (for the fine tuning) and 10 (different) people in the test set. The same test set is used for both cases (with and without fine-tuning). Tables 2, 3 summarize the verification experiments results.