Collaborative representation-based robust face recognition by discriminative low-rank representation

We consider $n$ original bases $\mathbf{X}=\left[\boldsymbol{x}_{1},\boldsymbol{x}_{2},\ldots,\boldsymbol{x}_{n}\right]\in\mathbb{R}^{m\times n}$ collected from $N$ different classes, and $m$ is the dimension of each base. Class $i$ includes $n_{i}$ training images which denoted by $\mathbf{X}_{i}$. $\mathbf{X}$ can be rewritten as $\mathbf{X}=\left[\mathbf{X}_{1},\mathbf{X}_{2},\ldots,\mathbf{X}_{N}\right]$. When comes a new test sample $\boldsymbol{y}\in\mathbb{R}^{m}$, SRC aims to find a sparse linear representation coefficient vector $\boldsymbol{\alpha}\in\mathbb{R}^{n}$ so that $\boldsymbol{y}$ can be represented as $\boldsymbol{y}=\mathbf{X}\boldsymbol{\alpha}$. This approximation problem can be calculated via minimizing the following problem:

where $\lambda$ denotes a scalar constant, $\|\cdot\|_{2}$ denotes the $\ell_{2}$-norm, and $\|\cdot\|_{1}$ denotes the $\ell_{1}$-norm. Many algorithms, such as basis pursuit [33] and Homotopy [34] can be used to figure out the above $\ell_{1}$-norm minimization problem. The test sample $\boldsymbol{y}$ should lie in the space spanned by the training samples from the correct class. Once we get the solution $\hat{\boldsymbol{\alpha}}$ of (1), where $\hat{\boldsymbol{\alpha}}=\left[\boldsymbol{\alpha}_{1};\boldsymbol{\alpha}_{2},\ldots;\boldsymbol{\alpha}_{N}\right]$ and $\boldsymbol{\alpha}_{i}$ is the representation vector of $\hat{\boldsymbol{\alpha}}$ associated with class $i$, the test sample $\boldsymbol{y}$ can be recognized by the reconstruction error of each class, i.e.,

Based on the fact that face images from different subjects may have similar appearances, thus samples from the uncorrelated classes can participate in representing the test sample $\boldsymbol{y}$. A regularized least square method is developed with significantly lower complexity named CRC, which is formulated as,

where $\lambda$ is a balance factor. CRC can offer improvements in decreasing computational complexity by using the $\ell_{2}$-norm-based model. A closed-form solution $\hat{\boldsymbol{\rho}}=\left(\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}\right)^{-1}\mathbf{X}^{T}\boldsymbol{y}$ can be derived by solving (3), in which $\left(\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}\right)^{-1}\mathbf{X}^{T}$ can be pre-calculated which leads to the fast computation speed of CRC. In the classification stage, the regularized residuals $\boldsymbol{e}_{i}=\left\|\boldsymbol{y}-\mathbf{X}_{i}\hat{\boldsymbol{\rho}}_{i}\right\|_{2}/\left\|\hat{\boldsymbol{\rho}}_{i}\right\|_{2}$ is used to classify the test image $\boldsymbol{y}$ by utilizing the discrimination information contained in $\left\|\hat{\boldsymbol{\rho}}_{i}\right\|_{2}$, where $\hat{\boldsymbol{\rho}}_{i}$ is the coefficient vector associated with class $i$. Finally, the test sample is designated to the class that has the least regularized residual.

Low-rank matrix recovery technique is used in our proposed method to recover a clean data matrix, so we investigate its formulation for the purpose of completeness. The corrupted training samples $\mathbf{X}$ can be decomposed into $\mathbf{D}+\mathbf{E}$ by RPCA [24], in which $\mathbf{D}$ is the clean low rank matrix and $\mathbf{E}$ is the associated sparse error matrix. The rank of matrix $\mathbf{D}$ is minimized by RPCA, meanwhile, $\|\mathbf{E}\|_{0}$ is reduced for the sake of deriving the low-rank approximation of $\mathbf{X}$. The original formulation of RPCA is formulated as,

where $\|\cdot\|_{0}$ denotes the $\ell_{0}$-norm. Eq. (4) is NP-hard as well as highly nonconvex, it is relaxed by replacing the $\ell_{0}$-norm with the $\ell_{1}$-norm and the rank function with the nuclear norm. The new optimization problem is more tractable as follows:

The limitation of RPCA is that it assumes that all the column vectors in $\mathbf{X}$ are drawn from a single low-rank subspace [26]. This hypothesis is not general and reasonable because face images usually reside on a union of multiple subspaces. A modified rank optimization problem in LRR is presented by Liu et al. [25, 26] defined as follows:

where the representation matrix of $\mathbf{X}$ is denoted by $\mathbf{Z}$ and $\|\cdot\|_{l}$ is a certain regularization strategy for expressing different corruptions. The inexact augmented Lagrange multipliers (ALM) algorithm [35] is employed to efficiently solve the above optimization problem (6). After the optimal solution $\mathbf{Z}^{*}$ is obtained, the recovered clean data matrix can be acquired from the corrupted data matrix $\mathbf{X}$ by $\mathbf{XZ}^{*}$.

The low-rank matrix recovery techniques can be utilized to improve the recognition accuracy of CRC because the problems brought by corrupted training samples can be alleviated. The recovered clean dictionary with a better representation ability can be obtained as $\mathbf{XZ}^{*}$ from the original matrix $\mathbf{X}$ by solving (6). One fact is that face images from different people share similarities due to the location of eyes, mouth, etc., so the drawback is that discriminative information is not contained by $\mathbf{XZ}^{*}$ and it is not suitable for classification. Inspired by [36], we propose a DLRR scheme for matrix recovery, different from LRR, the incoherence between different subjects in $\mathbf{XZ}^{*}$ is promoted. Consequently, the introduction of such incoherence would make the derived low-rank matrix from different subjects as independent as possible. The discrimination capacity is improved while the commonly shared features are suppressed.

A set of training images with corruptions are set as the original data matrix $\mathbf{X}=\left[\mathbf{X}_{1},\mathbf{X}_{2},\ldots,\mathbf{X}_{N}\right]$, where $\mathbf{X}_{i}$ is composed of the training images from class $i$. As mentioned above, the data matrix $\mathbf{X}$ can be decomposed into a low-rank matrix $\mathbf{D}=\left[\mathbf{D}_{1},\mathbf{D}_{2},\ldots,\mathbf{D}_{N}\right]$ and the sparse error matrix $\mathbf{E}$ by the formulation in (6), where $\mathbf{D}_{i}=\mathbf{X}_{i}\mathbf{Z}_{i}$ represents the clean data matrix from subject $i$. A regularization term $\left\|\mathbf{D}_{j}^{T}\mathbf{D}_{i}\right\|_{F}^{2}$ which sums the Frobenius norms of each pair of low-rank matrix $\mathbf{D}$ is added to the original LRR formulation to improve the independence of different classes. If the value of $\left\|\mathbf{D}_{j}^{T}\mathbf{D}_{i}\right\|_{F}^{2}$ is as small as possible, then the between-class independence can be achieved in our method. The new optimization problem is formulated as follows,

where $\lambda$ is scalar parameter. The last term promotes the structural incoherence of different subjects, which is penalized by the scalar parameter $\eta$ balancing the low-rank matrix decomposition and discriminative features. When draw a comparison between (7) and (6), the regularization term is utilized to provide improved discriminative ability, which can enforce more training samples from the correct subject to represent the test samples. We use the $\ell_{2,1}$-norm to encourage the columns of error term matrix $\mathbf{E}$ which represents extreme noise to be zero, the other extra regularization is unnecessary to be used on $\mathbf{E}$ since $\mathbf{E}$ is sparse. As a result, our new formulation (7) can fully explore the discrimination capacity contained in the original corrupted training images.

The discriminatory ability of the DLRR scheme in classifying face samples from different subjects is illustrated in Fig. 2. Similar to former Eigenfaces and CRC-based methods, we use the training and test face images from two different subjects in the AR face database and then project them onto the first two eigenvectors of the covariance matrix of the original data matrix $\mathbf{X}$ as shown in Fig. 2 (a), then the same data is projected onto the subspace which is derived by $\mathbf{XZ}^{*}$ without structural incoherence, the result is shown in Fig. 2 (b), and Fig. 2 (c) is the result of our proposed method. It is obvious that the distinction between the data projected onto $\mathbf{XZ}^{*}$ with and without structural incoherence are both improved compared with Fig. 2 (a). However, compared Fig. 2 (b) with Fig. 2 (c), we can find that the within-class scatter in Fig. 2 (c) is smaller than that of in Fig. 2 (b), and thus a better discriminative ability can be obtained by using our proposed approach. We also choose other images from three different classes in the Extended Yale B face database to do the same experiment as described above, as shown in the second row of Fig. 2. We plot their corresponding 2D subspaces spanned by the first two eigenvectors in Figs. 2 (d), (e) and (f), respectively. It is worth noting that our method, i.e.Fig. 2 (f), exhibits desirable discrimination capacity, while the distinction between the data projected onto the original data matrix $\mathbf{X}$ and the low-rank matrix $\mathbf{XZ}^{*}$ are observed to be degraded. Hence, better representation ability can be achieve by utilizing our derived LR matrix.

Our proposed optimization problem (7) is solved by ALM [35] in this section. Firstly, we introduce an auxiliary variable $\mathbf{J}_{i}$ to make the optimization problem (7) solvable, the new equivalent optimization problem is given as follows,

The transformed augmented Lagrangian function is formulated as an unconstrained optimization problem,

where $\mathbf{Y}_{1}$ and $\mathbf{Y}_{2}$ are Lagrangian multipliers, and $\mu>0$ is used as a penalty parameter. The optimization problem (9) can be rewritten by some simple algebra as follows,

where $f\left(\mathbf{Z}_{i},\mathbf{J}_{i},\mathbf{E}_{i},\mathbf{Y}_{1},\mathbf{Y}_{2},\mu\right)=\frac{\mu}{2}\left(\left\|\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}-\mathbf{E}_{i}+\frac{\mathbf{Y}_{1}}{\mu}\right\|_{F}^{2}+\left\|\mathbf{Z}_{i}-\mathbf{J}_{i}+\frac{\mathbf{Y}_{2}}{\mu}\right\|_{F}^{2}\right)$

The variables $\mathbf{Z}_{i}$, $\mathbf{J}_{i}$, $\mathbf{E}_{i}$ could be iteratively updated. Two variables are fixed at each iteration to update the remaining one. The detailed updating schemes are showed as follows in each step.

1. 1.

   Updating $\mathbf{Z}_{i}$

   Updating $\mathbf{Z}_{i}$ by minimizing $L\left(\mathbf{Z}_{i},\mathbf{E}_{i}^{k},\mathbf{J}_{i}^{k},\mathbf{Y}_{1}^{k},\mathbf{Y}_{2}^{k},\mu_{k}\right)$ is equivalent to minimizing the following unconstrained minimization function for $\sigma=\left\|\mathbf{A}_{i}\right\|_{2}^{2}$.

   $\arg\min\left\|\mathbf{Z}_{i}\right\|_{*}+\left\langle\mathbf{Z}_{i}-\mathbf{Z}_{i}^{k},\nabla_{\mathbf{Z}_{i}}f\left(\mathbf{Z}_{i}^{k},\mathbf{J}_{i}^{k},\mathbf{E}_{i}^{k},\mathbf{Y}_{1}^{k},\mathbf{Y}_{2}^{k},\mu_{k}\right)\right\rangle$ $+\frac{\mu\sigma}{2}\left\|\mathbf{Z}_{i}-\mathbf{Z}_{i}^{k}\right\|_{F}^{2}$ where $\nabla_{\mathbf{Z}_{i}}f\left(\mathbf{Z}_{i}^{k}\right)=\mu\left[-\mathbf{X}_{i}^{T}\left(\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k}-\mathbf{E}_{i}^{k}+\frac{\mathbf{Y}_{1}^{k}}{\mu_{k}}\right)+\left(\mathbf{Z}_{i}^{k}-\mathbf{J}_{i}^{k}+\frac{\mathbf{Y}_{2}^{k}}{\mu_{k}}\right)\right]$

   The above problem has the following closed-form solution,

   $$\mathbf{Z}_{i}^{k+1}=\arg\min\left\|\mathbf{Z}_{i}\right\|_{*}+\frac{\mu\sigma}{2}\left\|\mathbf{Z}_{i}-\left(Z_{i}^{k}-\frac{\nabla_{\mathbf{Z}}f\left(\mathbf{Z}_{i}^{k}\right)}{\mu\sigma}\right)\right\|_{F}^{2}$$

   (11)

2. 2.

   Updating $\mathbf{J}_{i}$

   Now by fixing $\mathbf{Z}_{i}$, $\mathbf{E}_{i}$, $\mathbf{Y}_{1}$ and $\mathbf{Y}_{2}$, we optimize the variable $\mathbf{J}_{i}$ for class $i$. The updating scheme of $\mathbf{J}_{i}$ is as follows:

   $\mathbf{J}_{i}^{k+1}=\arg\min_{\mathbf{J}_{i}}L\left(\mathbf{Z}_{i}^{k+1},\mathbf{E}_{i}^{k},\mathbf{J}_{i},\mathbf{Y}_{1}^{k},\mathbf{Y}_{2}^{k},\mu_{k}\right)$

   Then the solution to the above problem can be solved by solving the partial derivatives of $L$ w.r.t. $\mathbf{J}_{i}$ and setting it to be zero, and the solution is given by,

   $$\mathbf{J}_{i}^{k+1}=\left(\eta\sum_{j=1,i\neq j}^{N}\mu\mathbf{B}_{j}^{T}\mathbf{B}_{j}+\mu_{k}\mathbf{I}\right)^{-1}\left(\mu_{k}\mathbf{Z}_{i}^{k+1}+\mathbf{Y}_{2}^{k}\right)$$

   (12)

   where $\mathbf{B}_{j}=\mathbf{Z}_{j}^{T}\mathbf{X}_{j}^{T}\mathbf{X}_{i}$

3. 3.

   Updating $\mathbf{E}_{i}$

   By fixing $\mathbf{Z}_{i}$, $\mathbf{J}_{i}$, $\mathbf{Y}_{1}$ and $\mathbf{Y}_{2}$, we update the error matrix $\mathbf{E}_{i}$ for subject $i$ as follows,

   $$\begin{array}[]{l}{\mathbf{E}_{i}^{k+1}=\min_{\mathbf{E}_{i}}\lambda\left\|\mathbf{E}_{i}\right\|_{2,1}+\left\langle\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}-\mathbf{E}_{i},\mathbf{Y}_{1}^{k}\right\rangle+\frac{\mu_{k}}{2}\left\|\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}-\mathbf{E}_{i}\right\|_{F}^{2}}\\ {=\min_{\mathbf{E}_{i}}\lambda\left\|\mathbf{E}_{i}\right\|_{2,1}+\frac{\mu_{k}}{2}\operatorname{tr}\left[\left(\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}-\mathbf{E}_{i}\right)^{T}\left(\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}-\mathbf{E}_{i}\right)\right.}\\ {\left.+\frac{2\mathbf{Y}_{1}^{k}}{\mu_{k}}\left(\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}-\mathbf{E}_{i}\right)^{T}\right]}\\ {=\min_{\mathbf{E}_{i}}\frac{\lambda}{\mu_{k}}\left\|\mathbf{E}_{i}\right\|_{2,1}+\frac{1}{2}\left\|\mathbf{E}_{i}-\left(\mathbf{X}_{i}-\mathbf{X}_{i}\mathbf{Z}_{i}^{k+1}+\frac{\mathbf{Y}_{1}^{k}}{\mu_{k}}\right)\right\|_{F}^{2}}\end{array}$$

   (13)

Algorithm 1 summaries the whole detailed procedures for solving the optimization problem (10).

An example is given in Fig. 3 to intuitively display the recovery results of DLRR for matrix recovery, the corrupted images are successfully separated into the recovered clean images and the error images. Some training samples from one subject in AR database with illumination variations, pose changes, and occlusions are shown in Fig. 3 (a). The recovered clean samples and the corresponding sparse errors are shown in Figs. 3 (b) and (c), respectively.

In the testing phase, to handle the possible occlusion variations appeared in the test samples, which could degrade the performance of CRC. Motivated by Bao et al. [37], we try to find a low-rank projection matrix which can project the new corrupted samples onto their corresponding underlying subspaces.

After the original corrupted samples $\mathbf{X}=\left[\mathbf{X}_{1},\ldots,\mathbf{X}_{N}\right]\in\mathbb{R}^{m\times n}$ are successfully separated into the recovered matrix $\mathbf{Y}=\left[\mathbf{X}_{1}\mathbf{Z}_{1},\ldots,\mathbf{X}_{N}\mathbf{Z}_{N}\right]\in\mathbb{R}^{m\times n}$ and the sparse error matrix. The matrix $\mathbf{Y}$ can be seen as the set of principal components obtained from the matrix $\mathbf{X}$, so a linear projection $\mathbf{P}$ can be studied between $\mathbf{X}$ and $\mathbf{Y}$. Then any data sample $\boldsymbol{x}$ can be projected onto its underlying subspace to get the recovery results as $\mathbf{P}\boldsymbol{x}$. Based on the assumption that the recovery result is considered to be drawn from a union of multiple low-rank subspaces, we could hypothesize that $\mathbf{P}$ is a low-rank matrix and the optimization problem can be formulated as follows,

As mentioned in Section 2, a convex relaxation of the optimization problem 14 is proposed by replacing the rank function with the nuclear norm which can decrease the computational complexity. The convex optimization problem is formulated as,

$\mathbf{P}^{*}=\mathbf{YX}^{+}$ is formulated as the uniqueness of the minimizer for the above problem (15) under the hypothesis that $\mathbf{P}\neq 0$ and $\mathbf{Y}=\mathbf{PX}$ has feasible solution, where $\mathbf{X}^{+}$ is the pseudo-inverse of $\mathbf{X}$. The new test sample $\boldsymbol{y}$ can be recovered by $\mathbf{P}^{*}\boldsymbol{y}$ after obtained the optimal solution $\mathbf{P}^{*}$.

We outline the detailed procedures of collaborative representation-based classification by discriminative low-rank representation in Algorithm 2.

The AR database [38] contains over 4,000 frontal images from 126 subjects. In our experiments a subset from the AR database that contains 50 male subjects and 50 female subjects are used. There are 26 face images available for each subject, divided into two sessions, under the variations of expression, illumination, and disguise. In each session, there are 7 clean images with illumination and expressions variations, 3 images in sunglasses and the remaining 3 images in scarves disguise. All face images are cropped to 165$\times$120 pixels and then converted into grayscale before training and testing. Some images from the first subject in the AR database are shown in Fig. 4 (a). We consider the following three scenarios to validate the performance of DLRR-CR as in [21].

1. 1.

   Sunglasses: We consider the situation in which training and test images corrupted by sunglasses simultaneously. The presence of sunglasses produce about 20% occlusion of the frontal face image. From session one, we use eight training images, 7 neutral images plus one image with sunglasses. We use twelve test images, all non-occluded images from session two plus the rest of the face images with sunglasses.

2. 2.

   Scarf: We consider the situation in which training and test images corrupted by disguise simultaneously due to scarf, which occlude about 40% of the frontal face image. A similar choice of training and testing set is applied as above. From the first session, we use 7 neutral plus one with scarf for training. 7 non-occluded from the other session plus the remaining images with scarves for testing.

3. 3.

   Sunglasses+Scarf: In the final situation, we consider the most challenging case which training and test images occluded by mixed corruption due to sunglasses and scarves. From the first session, we use 7 neutral plus one with sunglasses and the other with a scarf for training. 7 non-occluded from the other session add the remaining images for testing.

For fair comparison, the feature dimensions are reduced to the same size for all methods. The regularization parameters in our method are set as $\beta$=1.1, $\eta=0.001$ and $\lambda$=0.02. The recognition accuracy of the three situations are given in Tables 1-3. From the results we can see that our method almost exceeds the other methods in each situation, which demonstrates the superiority of our method over the other approaches in dealing with real disguise.

The Extended Yale B database [40] includes 2414 frontal face images for 38 subjects, each subject has 64 face images obtained under different laboratory-controlled lighting conditions. The face images are cropped to a size of 192$\times$168 pixels and normalized in advance. Some exampe face images from the the extended Yale B database are shown in Fig. 4 (b). Firstly, 16 face images are randomly selected from each individual for training, and the remaining images for testing. Secondly, 32 face images are randomly selected from each subject for training, and the remaining images for testing. The eigenface feature dimensions are set to the same as in the experiments in the AR database. The regularization parameters used in DLRR-CR are set as $\beta$=1.1, $\eta$=0.001. Depending on feature dimension, the parameter $\lambda$ ranges from 0.004 to 0.1 for the two cases. All experiments run 5 times and the averaged accuracy is reported for performance evaluation shown in Tables 4-5.

From the results in Tables 4-5, for each dimension, DLRR-CR outperforms the other methods, this indictes our method can handle the problem of changes in illumination and expression. It should be noted that in the step 5 of Algorithm 2, the original training samples are used to reduce dimensions. The main reason is that we have already learned a desirable PCA subspace by the derived clean dictionary with discrimination and this subspace would not be too sensitive to sparse errors. The second reason is that CRC represents test sample collaboratively by all classes, a small proportion of corrupted training samples will have a small influence and there are also abundant images taken under well controlled settings which can participate in representing the test sample.

In this subsection, we consider the scenario in which both the training and test images are corrupted due to artificial occlusion. The face images from the Extended Yale B database are used to investigate the robustness of all competing approaches. In the first kind of setting, we randomly select 10% of all the images, then we randomly select pixels of these images, and these pixels are replaced by a random value in the range of [0,1]. In the second kind of setting, to examine the robustness of block occlusion, we also randomly selected 10% of all the samples from the database, then we randomly select square blocks of these images, and these square blocks are replaced by an unrelated image. Some representative examples of images with these two kinds of artificial occlusion are shown in Fig. 5. The percentage of corrupted samples in both situation are set to 10% and 20%.

When the artificial occlusion is added, 32 images are randomly selected from each class for training and the rest for testing. Now the regularization parameters are set as $\beta$=1.1, $\eta$=0.001 and depend on different feature dimensions, $\lambda$ ranges from 0.005 to 0.1. We investigate the classification accuracy with the eigenface feature dimensions set as 50, 100 and 300. We run 5 times of all experiments and the average accuracy is recorded. Table 6 shows the recognition accuracy of all seven algorithms for the two kinds of percentages of pixel corruptions. Specifically, our method achieves the best recognition performance, which are higher than those of the other methods. The recognition accuracy of block occlusion is shown in Table 7. We can also see that the performance gains of our algorithm is significant.