Two-Dimensional Dynamic Fusion for Continuous Authentication

An overview of the proposed continuous authentication framework is given in Figure 2. At the core of the framework are a classifier scheduling algorithm and a corresponding fusion algorithm that, in combination, implements our context-aware dynamic multibiometric fusion. The classifier scheduling algorithm will dynamically activate classifiers based on probability estimates of each classifier successfully authenticating a genuine user in the given context determined via environmental sensor outputs³³3These environmental sensors should measure environmental variables relevant to estimating the efficacy of the used biometric classifiers and should be sufficiently lightweight to be continuously running in the background. In practice, these will be, e.g., a light sensor detecting the level of light, a microphone detecting the noise level, and similar types of sensors.. The set of classifiers to be activated is chosen as the set which simultaneously has a sufficiently high combined authentication probability and the lowest recourse cost (see Section 2.2 for a detailed description). The corresponding fusion algorithm is based on the history of obtained classifier scores and will compute a fusion score $\beta$ based on the classifier scores obtained within a given (context-dependent) time frame which we denote the authentication window. In other words, the fusion algorithm implements a two-dimensional fusion approach combining multi-modal and multi-sample fusion as illustrated in Figure 2. Note that the classifier authentication scores available to the fusion algorithm are dynamically decided by the classifier scheduling algorithm, and hence, the classifier scheduling algorithm implicitly decides the type of fusion done in the fusion algorithm. The fusion algorithm is described in Section 2.3.

Finally, based on a comparison between the fusion score $\beta$ and an authentication threshold $th_{\beta}$, the device is either kept in an unlocked state or locked; i.e., the user is required to authenticate via an explicit authentication method (e.g., pin code, fingerprint scan, etc.). In the following subsections, the details of the framework will be described.

The classifier scheduling algorithm will decide upon the best possible set of classifiers to activate based on an a priori probability estimate for successful authentication (true acceptance rate) for each classifier in a given context (e.g., the performance of facial recognition in different levels of light), and is parameterized by a probability threshold $th_{p}$. Furthermore, the algorithm is assumed to have access to the resource cost of activating a set of classifiers and the time required to obtain classifier scores (i.e., input capture and processing). This is captured via the following algorithms:

$\mathtt{AuthProb}(cid,c)$

: On input a classifier id $cid$ and context $c$, this algorithm returns an estimated true acceptance rate.

$\mathtt{Time}(cid)$

: On input a classifier id $cid$, this algorithm returns the time required to capture input from the classifier $cid$ and process this.

$\mathtt{Cost}(\mathcal{S})$

: On input a set of classifier ids $\mathcal{S}$, this algorithm returns a scalar indicating the total resource cost of capturing input from $\mathcal{S}$ and processing this.

To estimate the probability of successful authentication using a set $\mathcal{S}$ of classifiers, the activation scheduling algorithm heuristically assumes independence of classifier authentication success. More precisely, the set $\mathcal{S}$ is considered a valid candidate set if

i.e., if the combined a priori probability of successful authentication is above the given threshold $th_{p}$. The activation scheduling algorithm then simply selects the candidate set with the combined lowest resource cost as the set of classifiers to be activated.

As a fail-safe mechanism, the classifier scheduling algorithm will ensure that all classifiers have been activated at least once in an attempt to authenticate the user before a potential lock of the device. This is achieved by scheduling any classifier which would otherwise not have time to complete an authentication attempt. Specifically, let $t_{crit}$ be the critical time at which the device is locked if no additional authentication input is obtained, $t_{now}$ be the current time, $\Delta t_{crit}$ be the time period until $t_{crit}$, i.e., $\Delta t_{crit}=t_{crit}-t_{now}$, and $\Delta t_{delay}$ be the time between each invocation of the classifier scheduling algorithm (see Section 2.4 for how $t_{crit}$ and $\Delta t_{crit}$ are derived)⁴⁴4We use the notation $t$ to denote a time instant and $\Delta t$ to denote a time period/interval.. Then a classifier $cid$ is activated if $\Delta t_{cid}+\Delta t_{delay}\geq\Delta t_{crit}$, where $\Delta t_{cid}\leftarrow\mathtt{Time}(cid)$, regardless of the cost.

The classifier scheduling algorithm is shown as Algorithm 1. Note that if no valid candidate set is found, the classifier scheduling algorithm will default to activating all classifiers.

Our fusion approach is based on the notion of an authentication window $\Delta t_{window}$ within which a user must be successfully authenticated to be allowed continued use of the mobile device. The authentication window itself is derived from the current context⁵⁵5Letting the authentication window be context-dependent allows the continuous authentication system to implement a more flexible security policy, e.g., by letting the window length depend on whether the device is in a low-risk environment such as a company officer or personal home, or in a high-risk environment such as a public place, or whether the running application handles potentially sensitive information. These factors can be identified from context information like geolocation, connected/available wifi networks, and running application identification., and we use the following abstract function to capture this:

$\mathtt{AuthWindow}(c)$

Given context $c$, this algorithm returns an authentication window duration $\Delta t_{window}$.

The fusion algorithm is then implemented by deriving an overall confidence score $\beta$ of the user being present from all individual authentication results obtained within $\Delta t_{window}$. We will let $\mathcal{H}=\{\mathcal{H}_{cid}\}_{cid\in\mathcal{S}}$ denote the history of all classifier scores, where $\mathcal{H}_{cid}=\{(\alpha,t)\}$ denotes the list of scores $\alpha$ obtained at time $t$ for each classifier $cid\in\mathcal{S}$, and let $\mathcal{H}_{cid}[t>t^{\prime}]$ denote scores obtained at time $t^{\prime}$ or later (i.e., $\mathcal{H}_{cid}[t>t^{\prime}]=\{(\alpha,t)\in\mathcal{H}_{cid}:t>t^{\prime}\}$). As illustrated in Figure 2, the fusion algorithm will first fuse individual classifier scores via a sample fusion rule $\mathtt{SampleFusion}$ to obtain a score $\beta_{cid}$ for each classifier $cid\in\mathcal{S}$; i.e., letting $\Gamma_{cid}\leftarrow\{\alpha:(\alpha,t)\in\mathcal{H}_{cid}[t>t_{now}-\Delta t_{window}]\}$,

Then an overall score is derived via a classifier fusion rule $\mathtt{ClassifierFusion}$:

Since our approach is based on selectively activating classifiers, it is likely that input from only a few classifiers is available. On the other hand, as the scheduling algorithm only activates classifiers that are likely to authenticate a genuine user (in the given context), the available scores are likely to be the most reliable scores obtainable from the classifiers. Hence, in our concrete scheme, we apply a simple approach to $\mathtt{SampleFusion}$ and $\mathtt{ClassifierFusion}$. Specifically, after score normalization (e.g., z-score normalization [7]), we use the average classifier score:

where the average of an empty set is defined to be $\bot$, and

where any value $\bot$ is ignored in the average.

The fusion algorithm, using the general $\mathtt{SampleFusion}$ and $\mathtt{ClassifierFusion}$ rules, is shown as Algorithm 2.

To realize the framework in Figure 2, we use the authentication loop shown in Algorithm 3. This continuously obtains the context via the external algorithm $\mathtt{ExtractContext}$ and runs the classifier scheduling algorithm (Algorithm 1) and fusion algorithm (Algorithm 2). Based on the fused score $\beta$ from the latter, the algorithm updates the device state to be either locked or unlocked via a threshold decision:

where the threshold $th_{\beta}$ is a parameter of the framework⁶⁶6Here, we assume that scores are defined as similarities. We can also handle scores defined as distances by locking the device if $\beta>th_{\beta}$.. When the device enters a locked state, the user is required to explicitly authenticate, e.g., via fingerprint scan, PIN, or password, to use the device.

The authentication loop maintains the history $\mathcal{H}$ of classifier scores and updates this via the external algorithm $\mathtt{RetrieveLastClassifierResults}$, which is required to return a list of activated classifier results $\mathcal{R}=\{(cid,\alpha,t)\}$ that have become available since the last invocation, where $cid$ is the classifier id, $\alpha$ the classifier score, and $t$ the time at which the score was obtained. The history $\mathcal{H}$ is required for running $\mathtt{Fusion}$ and to determine the critical time $t_{crit}$ at which the device is locked unless additional authentication information is available. The critical time $t_{crit}$ is determined as the earliest future time where the device will be locked assuming no additional classifier scores are available from $t_{now}$ and is computed by considering the shortest period of time $\Delta t_{crit}$ the authentication window can be moved forward before the fusion score becomes lower than $th_{\beta}$; i.e., letting $t^{\prime}\leftarrow t_{now}-\Delta t_{window}$,

where $\mathtt{Fusion}(\mathcal{H}[t>t^{\prime}+\Delta t^{\prime}])$ denotes a fusion score obtained from scores at time $t^{\prime}+\Delta t^{\prime}$ or later. The critical time $t_{crit}$ is then $t_{crit}=t_{now}+\Delta t_{crit}$; see the illustration in Figure 3. We will denote this computation of $\Delta t_{crit}$ as $\mathtt{CriticalTime}(\mathcal{H},c)$.

After calculating the remaining authentication window $\Delta t_{crit}$, we run the classifier scheduling algorithm (Algorithm 1) to obtain a set $\mathcal{S}_{act}$ of classifier ids to be activated. Then, we activate $\mathcal{S}_{act}$ and calculate scores for them via the external algorithm $\mathtt{ActivateClassifiers}$.

The main authentication loop is shown as Algorithm 3.

From this dataset, we construct two simple classifiers:

Classifier 1

based on four ‘frontal’ images for each subject as enrollment images.

Classifier 2

based on four ‘profile’ images (two left and two right facing) for each subject as enrollment images.

The classifiers are implemented by using a pre-trained ResNet network with 29 convolution layers to extract a 128-dimensional feature vector for the enrollment images. Likewise, for any test image, a feature vector is extracted and the Euclidian distance to all enrollment images is computed, and finally, the minimum distance is used as a similarity score. The scores for both classifiers are normalized by computing the corresponding z-score [7].

We define two contexts for our experiment; a semi-frontal context (SF) based on the MPEG images and a profile context (P) based on the profile images. Specifically, we consider four images per subject (up, down, left, right) for the semi-frontal context and four images (two left, two right, different from enrollment) for the profile context. Of the 372 subject IDs in the datasets, 290 have sufficient data for our experimental setup.

For comparison, we consider simple max and sum rules [7], as well as CWMA [14]. The context-dependent weights required for CWMA are obtained by splitting the subject IDs into two sets; one used exclusively for the training of CWMA and the other for the actual experiment. The weights (values in $[0,1]$) are chosen to be the weights that minimize the equal error rate (EER) for the training set found via brute-force computation (precision $0.02$). Finally, for our approach, the training set is used to estimate the true acceptance rate of the classifiers at the EER point, and a threshold $th_{p}$ of $0.9$ was used. In this setting, our approach selects one classifier for each time instant⁷⁷7 We use additional separate input (same subject, the same context) when multiple samples are required..

Figures 4(a) and 4(b) show the DET curves for all of the compared fusion techniques in both of the considered contexts. In Tables 1 and 2, we also show the EER of each individual classifier and each fusion approach (we explain Classifier 3 and a multi-modal experiment in Section 3.2). The DET curves demonstrate a clear advantage for our approach when running at a high frequency (i.e., twice as often as the others). Note that assuming an equal cost of the classifiers, this result is achieved without the use of additional resources. We also note that in the ‘semi-frontal’ context, where a single classifier greatly outperforms the other, our low-frequency approach (using half the resources compared to the other) performs similarly to the max fusion rule (cf. Figure 4(a)), as our approach selects the best classifier.

For the experiment, we consider three classifiers:

Classifier 1

identical to Classifier 1 from Section 3.1.

Classifier 2

identical to Classifier 2 from Section 3.1.

Classifier 3

pre-trained voice recognition system, where each subject is enrolled according to the enroll-core list of the SITW dataset.

The last classifier is implemented via a pre-trained proprietary voice recognition system from FueTrek⁸⁸8https://www.fuetrek.co.jp/. Note that this system is not trained on the SITW dataset and is optimized to recognize speech in Japanese of longer length..

We define four contexts based on the used data corresponding to the four possible combinations of a semi-frontal or a profile image sample for face recognition, and a low noise (LN) or high noise (HN) speech sample for voice recognition. Unfortunately, there are very few voice subjects for which we have both a clean and noisy speech sample, and the total number of genuine samples is much lower compared to the possible number of genuine samples for face recognition. Hence, we adopt an approach based on random sampling. Specifically, to sample a genuine sample for a given context, we first randomly pick XM2VTS and SITW subject IDs, with the choice of SITW ID being restricted to the IDs with samples matching the context (i.e., low or high noise). Then we randomly pick genuine image and speech samples from all available genuine samples for the chosen IDs and context. Finally, we run the above three classifiers on the chosen samples to obtain the corresponding scores. We take the same approach for all contexts and simply use samples (image and voice) for a different randomly chosen ID to generate impostor samples. Our dataset contains a total of $152$ and $5800$ genuine speech and face samples, respectively.

As in the uni-modal experiment, we compare our approach to the max and sum fusion rules as well as CWMA, and derive the optimal weights for the latter via brute-force computation on $1000$ samples obtained via the above random sampling process. Lastly, for our approach, the classifier performance is determined via $1000$ training samples and a threshold $th_{p}$ of $0.9$ is used. Again, our approach selects one classifier for each time instant in this setting.

These figures show that when running at a medium frequency (2x), our approach performs on par with CWMA and the sum fusion rule despite the latter two being based on three classifier inputs. In other words, our approach achieves almost the same accuracy as these existing approaches with fewer score calculations. Running our approach at a high frequency (3x) yields a clear advantage in all considered contexts, which means that ours achieves higher accuracy with the same number of score calculations.

The results shown in Sections 3.1 and 3.2 clearly demonstrates an advantage of our context-aware two-dimensional dynamic fusion approach compared to existing approaches when considering the same number of score calculations. Furthermore, as shown in Section 3.2, our approach can achieve similar performance to existing approaches using fewer score calculations, which illustrates the better accuracy and performance trade-off offered by our approach with multi-sample fusion.

Note that the actual resource cost of score calculations may vary between classifiers and will depend on the underlying hardware platform. Furthermore, some classifier can potentially be optimized for a multi-sample setup⁹⁹9For example, facial recognition can be combined with face tracking, which is comparable light-weight, to optimize the required image processing and subject identification. while this might not be possible for other classifiers. However, overall, the number of score calculations remains a good platform-independent resource measure.