Non-Contrastive Learning-based Behavioural Biometrics for Smart IoT Devices

There is a vast body of work proposing various behavioural biometric modalities. Early work involved using typing patterns and touch gestures [3, 14, 15] while later modalities leveraged human physiology  [5, 12, 7, 8, 16, 17]. The authentication solutions generally involve building machine learning classifiers or signature similarity-based approaches [9]. More recent works use deep learning methods, given their broader success in other domains [18].

Other works in behavioural biometrics aimed to increase the training efficiency with class incremental learning [19] or improved label efficiency using few-shot learning [20] and transfer learning [21]. Similar efforts were also made in human activity recognition [22, 23].

In contrast, we propose to improve the label efficiency by using non-contrastive self-supervised learning. Non-contrastive learning leverages large volumes of unlabelled data to build label-efficient classifiers. To the best of our knowledge, our work is the first to use non-contrastive learning for behavioural biometrics.

Self-supervised learning (SSL) refers to a broader family of methods in which a model learns representations from unlabelled data using pretext tasks. The pretext task acts as a feature extractor for supervised learning tasks reducing the labelled data requirement. For example, in computer vision, a pretext task learning may train a model to predict whether an image is an original or an augmentation. In this way, the model learns the distinguishing features of the original image. The pretext model is then fine-tuned for a downstream task in a supervised setting with labelled data. Jing et al. [24] provide a survey of SSL methods.

Early work closely resembling modern SSL includes Bromley et al. [25], where the authors proposed the ”Siamese” neural network architecture for signature verification. However, due to excessive resource requirements, SSL didn’t receive much attention until their success in natural language processing. In 2013, Mikolov et al. [26] used self-supervised learning to introduce word2vec, which paved the way to powerful generative language models such as BERT [27], RoBERTa [28] and XLM-R [29].

Nonetheless, neither generative methods [30, 31, 32] nor discriminative approaches [33, 34, 35, 36] were successful in other domains such as computer vision due to high computational complexity [37]. In contrast, Siamese networks-based comparative methods have shown promising results in computer vision [37, 38, 39, 13].

The basic form of Siamese networks consists of two identical neural networks which take two views of the same input (i.e., a positive pair) and outputs embeddings that have a low energy (or high similarity) between them. To increase the similarity of the two views, the networks learn spatial or temporal transformation invariant embeddings. Despite many successful applications of Siamese Networks, collapsing networks (where the network converges to a trivial solution) limit their performance.

To overcome these limitations, contrastive learning methods  [37, 40, 41, 42, 43] used negatives to avoid collapsing by not only pulling positives towards each other but also by pushing apart negatives in the embedding space. An example is the SimCLR model [37]. However, contrastive learning requires large batch sizes  [37, 43], support sets  [41], or memory queues  [42, 44, 40].

As a result, non-contrastive learning methods, and in particular the SimSiam model  [13], emerged as a viable alternative. Non-contrastive learning generally involves clustering  [39, 45], momentum encoders  [38], and using a cross-correlation matrix between the outputs of two identical networks as the objective function  [46], to address collapsing networks. These methods avoid the use of negatives to overcome the limitation of contrastive learning whereby two positive pair samples can get pushed apart in the embedding space, consequently becoming a negative pair and harming the performance of the end task [47]. However, the SimSiam [13] outperforms other non-contrastive approaches without using complex training approaches such as momentum encoders. It emphasises the importance of stop-gradient to present an efficient and a simple solution to the collapsing networks problem.

While SSL has majorly contributed to natural language processing, computer vision, and speech processing, its feasibility has been explored in sensing and mobile computing [48]. Saeed et al. [23] introduced self-supervised learning for time-series sensor data by introducing augmentations that are compatible with time-series data. The authors used a multi-task SSL model to reduce the labelled training data requirement in Human Activity Recognition (HAR). Using ten labelled samples per class, the authors achieved approximately 88.8% of the highest score reached by conventional supervised learning. SimCLR and several other contrastive and non-contrastive SSL methods also have been assessed on HAR problems  [49, 50]. Others such as Wright and Stewart [51] and Miller et al. [10] explored the use of traditional Siamese networks to reduce the training data requirement of behavioural biometrics-based user authentication.

In contrast to these works, to the best of our understanding, we are the first to propose SimSiam [13]-based non-contrastive learning for behavioural biometrics to reduce the labelled data requirement. Our method neither uses negatives nor requires complex training approaches such as momentum encoders to avoid collapsing. We compare our approach with baselines including traditional supervised learning, transfer learning, data augmentation, and state-of-the-art multi-task learning [23] and show that it can outperform supervised learning and provide comparable performance to multi-task learning at lower amounts of labelled data.

Our approach is based on the SimSiam architecture proposed by Chen et al. [13]. Its architecture is a more simplified non-contrastive architecture that doesn’t use negative pairs or other complex approaches to avoid collapsing. SimSiam architecture consists of two twin networks that share weights, as illustrated in Figure 1.

The idea is to learn a good representation of inputs by solving the problem of increasing the similarity between a positive pair $(x_{i},x_{j})$. A positive pair consists of two randomly augmented versions of the same input sample $x$. That is;

Here $\tau$ is a function that generates a random augmentation each time it is called. Then the two versions are encoded using the encoder network $g(x;\theta_{g})$,

The encoder consists of a feature extractor $g_{fe}(x;\theta_{fe})$ and a projector $g_{p}(x;\theta_{p})$. That is;

The key idea of the projector is to convert the representation learnt by the feature extractor to a vector that can be used to calculate the similarity. Next, the encodings go through another predictor network $h(x,\theta_{h})$ before calculating the similarity.

The purpose of the predictor is to predict the average of the representation vector across all possible augmentations the network has seen [13]. Next, the model calculates the cosine similarity within the pairs $(p_{i},z_{j})$ and $(p_{j},z_{i})$.

Here, $\lVert.\rVert_{2}$ denotes the $l_{2}$ norm of a vector. The task of the SimSiam model is to increase the total similarity, $Sim(p_{i},z_{j})+Sim(p_{j},z_{i})$. To do that, at training time the symmetric negative cosine similarity loss as defined below is used.

Note that, applying the stopgrad operation is essential for the SimSiam architecture to work [13]. It considers one side of the network as constant when computing the gradients of the other side, to prevent gradients from backpropagating in that direction as shown in Figure 1. During the training process the parameters, $\theta_{fe}$, $\theta_{p}$ and $\theta_{h}$ are learnt.

After the pre-text training of the SimSiam model, we transfer the trained feature extractor $g_{fe}(x;\theta_{fe})$ to our downstream task of building a classifier as illustrated in Figure 2.

We introduce two modifications to make SimSiam architecture work for time series behavioural biometrics data and further improve its performance. They are based on the hypothesis that easier self-supervision tasks lead to learning useless features and such features do not hold any value for subsequent downstream tasks. Therefore, it is important to make the self-supervised learning part more challenging so that robust features are learned during pre-text training.

• •

  Shallow feature extractor networks - The original SimSiam architecture was designed for image data. Time series data of behavioural biometrics are less complex compared to images and as such, to avoid the model over-fitting on pre-text the task, we use shallow feature extractor networks. A shallow feature extractor makes the learning task more difficult and as a result, allows the building of better feature extractors.

• •

  Weight decay - Using weight decay to prevent over-fitting is common in any machine learning application. We add high weight regularisation to the feature extractor network to avoid overfitting and make the pre-text training more challenging.

Later, in Section VI-B we provide an analysis of how the performance improves with our modifications. During our experiments, we also came across another important finding about the predictor network. We found that a deeper predictor network can help improve the non-contrastive SSL process. We provide further analysis and an explanation as to why it is happening in Section VI-C.

We conduct experiments to demonstrate the effectiveness of non-contrastive SSL in behavioural biometrics under three scenarios that are commonplace in authentication settings.

• •

  Scenario 1 - This scenario represents what is usually encountered by anyone who is developing a large-scale behavioural biometrics solution. That is, it is possible to collect large volumes of unlabelled data. For example, mobile OS providers can collect unlabelled data streams such as touch patterns and gait patterns from a large number of users, in compliance with privacy regulations. However, only a limited amount of labelled data can be collected from a known set of users due to usability and privacy constraints, either in-house or explicitly obtaining customer consent.
  

  More specifically, for $N_{1}$ users, a large volume of unlabelled data $X_{U_{1}}$ is available. For different $N_{2}$ users, only a limited amount of labelled data $(X_{L_{2_{min}}},Y_{2})$ is available. The task is to build a classifier $f_{SSL}(x;\theta)$ to identify the user $y\in\{1,...,N_{2}\}$ given the input $x$, by using both unlabelled data $X_{U_{1}}$ and limited labelled data $(X_{L_{2_{min}}},Y_{2})$. Here $|X_{U}|\gg|X_{L_{min}}|$.
  

  Here, we use the unlabelled data from $N_{1}$ users $X_{U_{1}}$ to pre-train a SimSiam-base feature extractor. Next, we train a classifier network on top of the pre-trained feature extractor for $N_{2}$ users. We use the labelled data $(X_{L_{2_{min}}},Y_{2})$ to train the classifier. We fine-tune the learnt weights of the feature extractor while training the classifier network. The concatenated fine-tuned feature extractor and the classifier network creates the final classifier $f_{SSL}(x;\theta)$ (cf. Figure 2).
  

• •

  Scenario 2 - This scenario is similar to Scenario 1. However here an organisation is trying to build an in-house authentication system. As a result, again there is a large volume of unlabelled data and a limited about of labelled data, but for the same set of users in contrast to Scenario 1.
  

  That is, for $N$ users, only a limited amount of labelled data, $(X_{L_{min}},Y)$ is available while a large volume of unlabelled data $X_{U}$ is available. The task is to build a classifier, $f_{SSL}(x;\theta)$ to correctly identify the user $y\in\{1,...,N\}$ given an input data sample, $x$, by using the limited labelled data, $X_{L_{min}}$ and unlabelled data $X_{U}$. Again here $|X_{U}|\gg|X_{L_{min}}|$.
  

  Similar to Scenario 1, here also we first pre-train a SimSiam-based feature extractor using unlabelled data $X_{U}$ and build a classifier network on top of the feature extractor for $N$ users using available limited labelled data, $(X_{L_{min}},Y)$. During classifier training, feature extractor fine-tuning happens the same as Scenario 1. The concatenated fine-tuned feature extractor and the classifier network create the final classifier $f_{SSL}(x;\theta)$.
  

• •

  Scenario 3 - This is an incremental step from Scenario 2, where the organisation has collected labelled and unlabelled data for building the authentication system but has additional new users for whom only a limited amount of labelled data is available.
  

  That is, for $N_{1}$ users, a limited amount of labelled data, $(X_{L_{1_{min}}},Y_{1})$ and a large volume of unlabelled data, $X_{U_{1}}$ is available. For different $N_{2}$ users, only a limited amount of labelled data $(X_{L_{2_{min}}},Y_{2})$ is available. The task is to build a classifier, $f_{SSL}(x;\theta)$ to correctly identify the user $y\in\{1,...,(N_{1}+N_{2})\}$ from the combined set, given an input data sample, by leveraging both unlabelled and labelled data.
  

  Here, we first use unlabelled data from $N_{1}$ users $X_{U_{1}}$ to pre-train the feature extractor with the SimSiam architecture. Then, we build a classifier on top of the pre-learned feature extractor, for the combined set of $N_{1}+N_{2}$ users using both labelled datasets $(X_{L_{1_{min}}},Y_{1})$ and $(X_{L_{2_{min}}},Y_{2})$. Similar to previous scenarios, we fine-tune the learnt weights of the feature extractor while training the classifier network. The concatenated fine-tuned feature extractor and the classifier network create the final classifier $f_{SSL}(x;\theta)$.
  

We compare our non-contrastive SSL approach with multiple baselines. Below we provide a general overview of the different baselines we use. However, we highlight that not all baselines apply for all three scenarios.

1. 1.

   Supervised learning - We train a 1D CNN based on available labelled data. For example, for Scenario 1, we leverage the available limited labelled data, $X_{L_{min}}$, and train $f_{S}(x;\theta)$. We do all the required hyperparameter tuning such as finding optimal convolution kernel sizes, number of convolutional filters in a layer, depth of the network, learning rates, and weight regularisation constants to ensure that we leverage the full capability of the supervised learning approach.

2. 2.

   Data augmentation - Data augmentation is a default step in any deep neural network training process. It helps to increase the generalisability of the model as well as learn from limited labelled data to some extent. In this baseline, we augment available limited labelled training data using two methods, scaling, and noise addition and continue to train a supervised learning classifier, as usual, using both limited labelled data and augmented data.

3. 3.

   Multi-task self-supervised learning (MTSSL) - This is a self-supervised learning baseline, which can leverage unlabelled data compared to the above supervised learning approach. As a result, it is a closer baseline to our approach of non-contrastive SSL. Here, we first train a multi-task model with a common feature extractor using unlabelled data available in a given scenario. The common feature extractor is connected to several heads, each having a dedicated discriminative task. Each head is a binary classifier learning to discriminate whether an assigned augmentation is applied to a sample or not. Next, we build a classifier on top of the pre-trained feature extractor by adding extra fully connected layers.

4. 4.

   Transfer learning - Transfer learning is the most common approach to handling the lack of labelled data. Here, the deep neural network trained using labelled data is leveraged as a feature extractor to facilitate adding new users to an existing behavioural biometric system or building a new behavioural biometric system from less labelled data. During the transfer learning phase, several new layers are added to the previous feature extractor, corresponding to the new classification task. The entire model is then fine-tuned with the available limited training data.

5. 5.

   Transfer learning with data augmentation - Here, we do transfer learning together with data augmentation.

In Table I we summarise the research questions associated with the three scenarios and the baselines used in each scenario. We give further details of the implementations aspects of the baselines methods in Section IV-C.

To measure the performance of trained non-contrastive SSL-based user authentication systems and compare them against other baselines we use Cohen’s Kappa coefficient similar to [23]. Usually, accuracy is the most commonly used metric to evaluate the performance in a multi-class setting. Accuracy measures the agreement between two raters, here raters being the true label and predicted label.

However, accuracy can be misleading in some occasions especially if the trained model is biased to predict one class more accurately and another class less accurately. In contrast Cohen’s Kappa coefficient measures the agreement between two raters but discount the effect of agreement by chance. That is,

$P_{o}=\text{Probability of agreement (Accuracy)}$
$P_{e}=\text{Probability of agreement by chance}$

That is,

$N=\text{Number of Classes}$
$n_{total}=\text{No. of total predictions}$
$n_{true}^{i}=\text{No. of true labels of the $i^{th}$ class}$
$n_{pred}^{i}=\text{No. of predicted labels of the $i^{th}$ class}$

The highest possible Kappa Score is 1 indicating the best performance and it can have negative values at worse performances. Overall, we progressively increase the amount of labelled data available and compare the performance of different methods using the Kappa Score. We further discuss this in Section V.

To demonstrate the effectiveness of non-contrastive learning in behavioural biometrics, we use two datasets; MusicID [7] and MMI [52]. Both of these datasets are EEG datasets and have been used in behavioural biometric settings before. Note that, for the rest of the paper, a session refers to a single experiment of recording sensor readings in one sitting for one user.

• •

  MusicID - This dataset consists of brainwave data collected from 20 volunteers while they performed two tasks [7]; listening to a popular English song and listening to the individual’s favourite song. The participants wore a Muse brain sensing headset while listening to the music and kept their eyes closed. The dataset experiment was approved by the host institution’s Human Research Ethics Committee as mentioned in the original work. The duration of each task in a single session was 150s and the headset records samples at a rate of 2Hz, resulting a total of 300 readings per participant, per session. Data was collected from each participant over multiple sessions with the number of sessions per user varying between 12-30 (considering both the same song and favourite song sessions together). Each headset recording contains 24 readings; absolute brainwave values of alpha, beta, theta, delta, gamma, and raw EEG from the standard 4-channels of the Muse headset. That is, a single reading in this dataset is a 24 dimensional vector, $x_{i}\in\mathbb{R}^{24}$. We use 30 readings (i.e., 15 seconds of data) as a single input sample to the model. Consequently, one input to our model is $x\in\mathbb{R}^{30\times 24}$.
  

• •

  MMI - This is a publicly available dataset known as eegmmidb (EEG Motor Movement/Imagery database). It comprises of EEG signals obtained from 109 participants using the BCI2000 system [52]. Separate experiments were conducted where each participant carried out four different tasks, each for a two-minute duration and repeated three times. The tasks involved different combinations of opening and closing fists or feet based on the location of a certain target shown on a screen. Each participant also performed two one-minute baseline runs. The data was obtained in EDF+ format – consisting of 64 EEG signals and an annotation channel where the annotation channel indicates the participant’s activity. We randomly selected eight channels; 3, 12, 13, 18, 50, 60, 61, and 64 to reduce the memory requirements. For each channel, we filter the raw signal to get the alpha, beta, theta, and delta components and use the raw signal and the four filtered components as input features to our models. That is, a single reading $x_{i}\in\mathbb{R}^{40}$. The sessions are recorded with a $160Hz$ sampling rate. We use $800ms$ of readings making our inputs to the models $x\in\mathbb{R}^{128\times 40}$.
  

  Even though this dataset has been used for user authentication [8], the previous work only use a portion of the dataset, EEG-S, which only contains data from eight users. However, we, on the other hand, incorporate all 109 users in our experiments. Due to the high number of users and the fact that this dataset was not collected with authentication as a target application, the maximum kappa scores we could achieve for user authentication in the MMI dataset is less than the MusicID dataset.

To emulate the three scenarios described in Section III-B, we split each dataset into two parts; Dataset 1 and Dataset 2. Dataset 1 contains data of approximately 1/3 of the users of the total dataset and Dataset 2 contains the rest. We use these two datasets in different ways in the three scenarios. Note that in the following description, ”labelled data of Dataset 1/Dataset 2” refers to $(x,y)$ pairs coming from a dataset while ”unlabelled data of Dataset 1/Dataset 2” refers to only $(x,)$ values coming from a dataset, ignoring the user ID labels.

In Scenario 1, we use Dataset 1 as the unlabelled data $X_{U_{1}}$ coming from $N_{1}$ users. We use labelled data from Dataset 2 as the labelled data $(X_{L_{2_{min}}},Y_{2})$ coming from $N_{2}$ users who will be part of the authentication system. We progressively increase the amount of labelled data in $(X_{L_{2_{min}}},Y_{2})$ to assess the performance of our method and other methods as presented in Section V-A.

Since in Scenario 2 both the labelled and unlabelled data are associated with the same set of users, we use only Dataset 2. In Scenario 3 we use Dataset 1 as the dataset coming from the initial users $N_{1}$ who provide both unlabelled $X_{U_{1}}$ and labelled data $(X_{L_{1_{min}}},Y_{1})$. We use labelled data from Dataset 2 as the labelled dataset $(X_{L_{2_{min}}},Y_{2})$ coming from the new set of users $N_{2}$.

We use dedicated sessions for validation and testing. Reading training data is done with a 50 percent overlapping window similar to [7]. We do not use overlapped windows when reading validation and test data since it will artificially increase the performance metrics. We summarise the two datasets and the splits in Table II.

In Figure 5a and Figure 5b we show the performance of our SimSiam method and various other baselines for MusicID and MMI datasets, respectively. The overall kappa score is low for the MMI dataset (approximately between 80% - 85%) because it is a much noisier dataset not necessarily designed for user authentication (cf. Section IV).

Both the figures show that conventional supervised learning and data augmentation do not perform well because there is not enough labelled data to train them for the second set of users. For both the datasets, transfer learning works best (i.e., shows a high kappa score at a given percentage of data samples per user) followed by multi-task learning and our SimSiam approach. On the MusicID dataset, SimSiam’s performance is competitive with multi-task learning (MTSSL), and on the MMI dataset, up to until 20% of data samples per user, SimSiam is slightly worse, but catches up with multi task learning afterwards.

The high performance of transfer learning can be attributed to the use of labelled data from Dataset 1 during the pre-training phase of transfer learning. Here, SimSiam SSL still could compete with the performance of the transfer learning without using any labelled data from Dataset 1 which indicates the its capability in extracting features from unlabelled data.

Overall, our results show that non-contrastive SSL can indeed learn generic features from unlabelled data of a set of users to build a classifier for a totally different set of users using less labelled data. For instance, for the MusicID dataset, between 20% to 40% samples per user, the average kappa scores for SimSiam was 0.978. Within the same percentages of labelled samples per user, the average kappa scores of supervised learning and data augmentation were 0.879. As a percentage improvement this corresponds to 11% improvement over the baselines. The corresponding percentage increase of the MMI dataset was approximately 4%. Finally, though multi-task SSL performs slightly better than SimSiam it is computationally expensive as we explain in Section VII.

In Figure 6a and Figure 6b we compare the results for Scenario 2 for the two datasets. At lower amounts of labelled data two self-supervised learning approaches; SimSiam and multi-task learning (MTSSL) are performing better compared to supervised learning approaches. For example, for the MusicID dataset, when only 20% of labelled samples are used, both SimSiam and MTSSL have average kappa scores of 0.956 and 0.985, respectively while supervised learning and data augmentation approaches only result in kappa scores of 0.885, and 0.800. This difference drops in the MMI dataset, which can be expected since it is a much larger dataset compared to the MusicID dataset and when 20% of labelled samples are used, it is sufficient to train the supervised learning classifier.

Finally, similar to Scenario 1, it is noticeable that between the two self-supervised learning methods, the multi-task approach shows better performance than the non-contrastive SimSiam approach and the difference is more visible in MMI dataset compared to the MusicID dataset. Nonetheless, our results show that unlabelled data can be leveraged using non-contrastive SSL to build more label-efficient classifiers. For instance, the average performance improvements of SimSiam over supervised and data augmentation approaches were 10% and 4% for the MusicID and MMI datasets, respectively.

Figure 7a and Figure 7b present the results for Scenario 3. Overall, the results are similar to Scenario 2, with SimSiam and multi-task learning performing above the other baselines at lower amounts of labelled data. However, compared to Scenario 2, the performance gap between different methods is reduced in Scenario 3. The reason behind this is having access to a larger amount of labelled data in this scenario compared to the other two scenarios. Here we train the classifier for the total user set. That is, 20 users for MusicID and 109 for MMI. Even though the number of samples a user provides is similar to other two scenarios, when taken as a whole it creates a large labelled dataset. Supervised learning methods benefit from this data and reach a performance similar to self-supervised methods. Even then, multi-task SSL and SimSiam outperform other methods and on the MMI dataset SimSiam slightly outperforms multi-task SSL after 20% of samples per user.

As described in Section III, the SimSiam training process involves feeding the network with positive pairs (i.e., two augmented versions of the same input). As such, it is important to identify data transformation/augmentation methods that result in better self-supervised feature learning. Thus, we train the SimSiam network with different augmentation technique pairs and compare their performance. As mentioned earlier, we keep all other network parameters fixed and only change the augmentation technique pair. We evaluate the feature extractors under both the experiment settings, $Ex_{1}$ and $Ex_{2}$, for both the datasets and report the average kappa scores in Figure 8a and Figure 8b.

According to Figure 8a any data augmentation technique pair gives high kappa scores of over 0.95 for the MusicID dataset. This can be attributed to the smaller size of the MusicID dataset and to the matching experiment conditions of the data collection process, which was designed for authentication applications from the beginning. In contrast, as can be seen from Figure 8b two augmentation technique pairs do not result in high kappa scores for the MMI dataset. Also, there is a considerable variation in kappa scores between pairs. For example, the augmentation pair Permutation and Magnitude Warping results in only a kappa score of 0.3049 while the pair Drop and Time Warping results in 0.4902. This can be attributed to the MMI dataset being large and noisy. Nonetheless, this analysis justifies using more than two augmentation methods to train a better feature extractor in Section V. For example, when more augmentations are considered, the kappa scores for the MMI reaches close to 0.8 (cf. Section V).

As mentioned in Section III-A we did two modifications to the SimSiam learning process to make it more suitable for behavioural biometrics data and improve its performance. Here, we experimentally show how the two modifications we introduced; shallow feature extractor and weight decay, improve the self-supervised training process.

In contrast to the feature extractor which needs to be shallow for higher performance, our experiments found that the predictor network needs to be deeper for better performance. To show this effect, we conduct several experiments keeping all other parameters fixed and only changing the predictor depth. We present our results in Table VII. Predictor architectures are given in a comma delimited format - corresponds to the dimensions of the Dense layers, from left to right. For example, the sixth predictor architecture corresponds to the architecture illustrated in Figure 4.

As mentioned in Section III-A, the role of the predictor is to average the representation vector across all possible augmentations the network has seen. A deeper predictor can memorise more augmentations, consequently making the averaging more precise. During the training process the model compares the output of the predictor $p$ with the output of the encoder network $z$. where $p$ is the mean vector of several augmentations of the same sample and $z$ is the representation vector of a single augmented version of that sample. The task of the network is to make $p$ and $z$ similar. Since $p$ is an averaged vector, in order to make $z$ similar, the encoder network is forced to learn a representation that is common to all the averaged versions. If the predictor is shallow, it can only compare only a few versions of the sample. Therefore, deeper predictor networks can help to learn a more generalised representation.