Deep Active Learning with Crowdsourcing Data for Privacy Policy Classification

Given the number and length of privacy policies, much of the related work focuses on extracting important information from them and identify the related data practices. Watanabe et al. [39] use keyword extraction on privacy policies to identify non-compliance between mobile apps and their privacy policies. Wilson et al. [40] built and trained text classifiers using the OPP-115 Corpus, which consists of 115 website privacy policies labeled by legal experts. Using the same data corpus, Frederick et al. [18] presented a performance comparison among Logistic Regression (LR), Support Vector Machine (SVM), and Convolutional Neural Network (CNN) models, and has used this to also detect non-compliance in Android applications [48]. Harkous et al. [10] developed a multi-label classifier using CNN to label policies using major categories and smaller attributes. Elisa et al. [5] presented a solution to automatically assess the completeness of a policy using more complex algorithms such as Linear Support Vector Machines (LSVM). Zimmeck et al. [47] also created a labeled mobile app-specific privacy policy corpus, APP-350. To compensate the rarity of negative annotation labels, they introduced synthetic data by manually changing positive policy texts into negative samples. Finally, Elisa et al. [5] extracted texts from 64 privacy policies, and labeled them by a single annotator. Because Calpric leverages crowdsourcing to efficiently label samples, it is able to build a larger labeled dataset and achieve significantly greater accuracy using machine learning than previous work. In addition, while Zimmeck et al. [47] used synthetic training points to address class imbalance, Calpric overcomes the imbalanced training set by mining a large unlabeled dataset for potential negative samples and gets them labeled with active learning.

Although Zimmeck et al. [46] and Wilson et al. [41] also explore the crowdsourcing option, in both studies, labelers are required to read through the entire privacy policy and answer questions accordingly. In contrast, we pre-preprocess policies into segments that address the same data practice to reduce labeling effort, as suggested by Schaub et al. [28]. Whereas Wilson et al. [41] focus on methodology to increase the crowdsourcing productivity, such as highlighting the most relevant paragraphs in a privacy policy, we shift the focus and extend our work to classification of policy segments. Our study also differs from all prior published work on privacy policy classification because we integrate active learning algorithms on top of regular classifiers. Combined with the use of policy segments, Calpric is able to address the class imbalance issue by automatically querying more negative samples in the unlabeled training pool.

Active learning is an iterative training strategy where an initial model is trained in the normal fashion, and it is then allowed to select unlabeled training instances using a query strategy. In each active learning iteration, a number of selected labels are sent to a query oracle for labeling and the model is then updated with the newly labeled training points—we use Amazon mTurk as our query oracle. The model can be trained for an arbitrary number of such iterations until a pre-defined stopping point is reached.

In general, there are three active learning scenarios: membership querying synthesis, stream-based, and pool-based selective sampling. Pool-based selective sampling has been the most well-studied scenario, especially for text classification [38] and information extraction [32]. The major advantage of this method is that it evaluates and ranks the entire set of unlabeled training points before selecting the next one to label [30]. We select pool-based sampling because it is applicable to Calpric’s scenario as we have the entire unlabeled training set up front and it is the most effective and widely used sampling mode for text classification.

To our knowledge, this is the first paper that performs deep active learning classification on privacy policies. However, there are related studies using similar learning approaches in areas such as image analysis and classification [42, 9], and natural language processing (NLP). Zhang et al. [43] implemented active learning strategies on top of CNNs for sentiment analysis, whereas Shen et al. [34] investigated uncertainty-based active learning heuristic for sequence tagging on a newly proposed CNN-CNN-LSTM architecture. We build upon prior work and experiment with our PPWE-biLSTM and the fine-tuned BERT [7] classifiers, which are proven to be the state-of-the-art NLP models for text classification tasks [45, 12].

We scrape Android application metadata, including the link to the app’s privacy policy, from the Android Google Playstore. We then filter out broken or invalid privacy policy links. We only consider policies that are in HTML format, though we could have easily extended our preprocessing to handle less common formats such as PDF or raw text. Our web scraper is based on the dragnet’s pre-trained model [25]. The downloaded HTML is then passed through a sanitization pipeline, in which the irrelevant elements, including HTML tags, advertisement, and UI features (eg., navigation bar), are removed. We verify that the downloaded pages are actually privacy policies by checking for the presence of keywords such as “privacy policy” and “legal” as well as by excluding any documents with fewer than 50 words. Finally, we verify text language using the Python langdetect library [24] as the Calpric models currently only handle English text. Finally, we identify and remove duplicates, leaving 51,781 usable privacy policies.

Next, we extract policy segments, which will form the training points for Calpric. Policy segmentation includes the following tasks: keyword filtering, segmentation and pre-classification.

We first tokenize each privacy policy into a list of sentences using the Python NLTK library [1]. Calpric then filters the sentences using keywords corresponding to three private data categories: contacts, location and device ID. For example, we use keywords such as “email” and “phone number” to extract sentences that discuss the handling of private data in the contacts category and keywords such as “IP address” for the device ID category.

We then segment the privacy policies. The objective of segmentation is to include only the necessary sentences required to understand what the privacy policy is declaring with respect to the private data category. A typical example of a policy segment is as follows:

Personal information is data that can be used to uniquely identify or contact a single person. When you visit, download or upgrade our app or our products, we do not use this information explicitly. However, we may collect personal information to improve our services and deliver a better experience.

All sentences in the segment contribute to the meaning of the segment. For instance, without the first sentence, it is not clear that personal information refers to contact information. However, if the segment only contains the first two sentences, readers may be misled to provide a negative label to the segment. Finally, the sentence immediately following the snippet was “Please be aware that when you register and set up an account, you will at minimum have to download the Application onto your mobile device.” This sentence is not required to understand whether the application was collecting private contact information or not, and thus should not be included in the segment.

Segments are constructed in two steps. First, Calpric checks for keywords such as “include” or “for example”, as well as punctuation marks such as semicolons and question marks, which indicate that the previous sentence may contain information related to the current one.

Second, Calpric uses NLP algorithms to measure the similarity of consecutive sentences. As a part of the algorithm, similar to Harkous et al. [10], we create a domain-specific word embedding trained by the 52K downloaded policies on top of Fasttext [2] and use it to generate vector representations of sentences. While there are various general-purpose pre-trained word embeddings available, customized embeddings tend to achieve better performance for classification tasks [37]. As a Fasttext-based embedding that allows vector training on sub-words, the policy embedding is able to interpret words with spelling mistakes and, more importantly, captures the actual meaning of proper nouns, which usually consist of several words.

Calpric measures sentence similarities using the Word Mover’s Distance (WMD) [16], which evaluates how relevant the previous sentence is to the current one. Dias et al. [8] propose a threshold function that consists of the average and the standard deviation of the downhills depths, where downhills represent topic shifts in a document. To determine if there is a topic boundary between two sentences, our tool compares the WMD similarity of these sentences against a Segment Threshold (ST), which is calculated for every individual privacy policy; if the similarity is larger than the threshold, the two sentences are considered to be related. Therefore there should be no boundary between them.

ST is calculated as follows:

After segmentation, Calpric now has a set of unlabeled segments labeled by private data category $\mathbb{X}^{U}_{contact}$, $\mathbb{X}^{U}_{device}$, and $\mathbb{X}^{U}_{location}$. In total our 52K privacy policies produced 153K, 63K, and 38K segments for contacts, location, and device ID, respectively.

Amazon mTurk provides a platform for requesters to publish HITs, in the form of surveys to a market, which are accepted and performed by Turkers. Our Amazon mTurk HITs consist of two questions (see Appendix A). The first question confirms that the policy segment is a First Party Collection/Use, while the second question asks whether or not the policy segment claims to collect/use private data of that type. Labels for segments where Turkers answer “No” for the first question are discarded, as these segments are likely not relevant to data collection. For the remaining labels, the answer to the second question is converted to a positive label if Turkers answer “Yes” and a negative label if Turkers answer “No”.

Because some Turkers may incorrectly label other policy segments, each survey is sent to five Turkers and we measure the level of agreement between the Turkers for each segment, and only accept labels where there is sufficient agreement. In addition, some privacy policy segments are inherently ambiguous and open to interpretation. Because they are a vehicle for obtaining consent, privacy policies should be understandable by people with no legal training. As a result, if regular people cannot agree on the meaning of a policy segment, then it has no definite meaning, and thus is not a useful training point for Calpric.

Our final survey questions were selected from a set of four survey variants we had constructed. To determine the best one, we computed the Alignment Rate (AR) for each survey variant. To calculate AR, we first need to introduce several definitions. Agreement Percentage (AP) refers to the percentage of Turkers labeled a segment the same way. For instance, if five workers select Yes for a segment, two select No, and yet another three others select Other, the AP will be $5/(3+2+5)=50\%$. To filter out low-confidence labels, we pre-define a threshold above which the AP must fall for the label to be considered reliable (i.e., the segment is not ambiguous and the Turkers labeled the segment correctly). We call this threshold the Acceptance Threshold (AT) and default it to 80% in this study. We refer labels that pass the AT as Aligned Labels. AR is thus the number of aligned labels extracted from segments over the total number of segments published for labeling. From this we can see that when used on the same set of segments, and run over a sufficiently large number of Turkers, a survey variant that achieves a higher AR is likely more consistently interpreted by the Turkers and will thus yield fewer erroneous labels. We note that this simple AR metric achieves a similar result as more complex measures of correlation of internal consistency as our labels are binary (collect or no-collect).

We measure AR for our variants by preparing three batches of questions, with $10$ different policy segments per batch. We randomly select the segments from the APP-350 dataset, which were labeled by skilled labelers and we consider them as $100\%$ reliable. Using four different versions of survey questions, we calculate the AR on $30$ policy segments drawn from the APP-350 dataset, with each segment labeled by five Turkers. The AR varies from $45\%$ to $73\%$, with the average labeling accuracy between $70\%$ to $75\%$ for all versions. We select the version with the highest AR for all future crowdsourcing tasks.

Because we create training labels by crowdsourcing instead of legal experts, we need to ensure label reliability and reduce data noise as much as possible. Previous studies show that there should be an emphasis on quality control when dealing with crowdsourcing tasks because they may have varying degrees of skill or may not pay full attention when performing HITs [5]. We introduced a set of quality controls to validate crowdsourcing workers and filter out low-quality data.

Because crowdsourcing holds the promise of providing more labeled data than previous studies, we are able to use more complex, higher capacity models than previous works that which used models such as LR and SVM. Traditional CNNs experience the issue of long-distance dependency in text classification, while recent NLP studies solve this problem by introducing RNNs, LSTM, and BERT models. These models have been proven to achieve a better overall performance in text classification [19, 45, 7].

A commonly used, state-of-the-art language model is BERT, which can be fine-tuned for various tasks including text classification. However, because active learning is an iterative training algorithm, an important factor is the retraining time of the model that is selected. While accurate, BERT suffers from slow training due to its complexity and the number of parameters involved, despite it being an application of transfer learning. Joselson and Hallen [12] evaluate a fine-tuned BERT model and a customized biLSTM classifier for sentiment analysis, and show that the former model requires hours more training while achieving similar performance ($69\%$ vs. $71\%$). As a result, developed our own model based on biLSTM, which we call PPWE-biLSTM.

The model is implement with bidirectional LSTM layers and a customized privacy policy word embedding trained on top of the general-purpose Fasttext embedding. Our LSTM nodes are bidirectional, so two recurrent layers in opposite directions serve as a platform for training in the past and future of a specific time frame. The PPWE-biLSTM used in our experiments consists of a hidden layer with $100$ densely connected memory units, as the structure is proved to achieve a lower perplexity than regular stacked models [13, 11]. We use leaky ReLUs [21] and dropouts [35] of $0.1$ to prevent over-fitting while sustaining the weight updates to avoid vanishing gradient problems in the propagation process [21]. We apply a sigmoid activation function to the model output for binary prediction. We use binary cross entropy loss and ADAM optimizer [14] to find model weights. We used BATCH_SIZE of $20$ and EPOCHS = 4. We apply the same vectorization method with our customized word embedding for the ML component. An evaluation of PPWE-biLSTM vs. BERT is included in Section VII-C. We use the above model setup as default for all experiments, unless specified otherwise.

Recall that we decide to investigate pool-based sampling because we have the entire unlabeled training set available and it is the most effective and widely used sampling mode for text classification. The detailed setup is described in the following sections, including querying strategies (Section VI-B1), alignment threshold (Section VI-B2), and re-labeling strategies (Section VI-B3).

As mentioned, we focus on policy segments for First Party Collection/Use, with data types being geographical location, device information, and contact information. We extracted these labels from APP-350 and OPP-115, and grouped them into the same categories. We analyzed the two corpora and summarized the results in Tables III and IV. The numbers $115$ and $350$ in OPP-115 and APP-350 indicate the respective numbers of privacy policies covered by each dataset. The number of policies covered in the corpus is different from the number of segments with useful labels.

We observe that the number of segments of interest for APP-350 is much larger than that of OPP-115. Note that the web-based OPP-115 dataset is more fine-grained and covers a larger set of categories, whereas the mobile application policies in APP-350 tend to be simpler. Policy segments in OPP-115 are mostly short paragraphs while APP-350 has a smaller average length of words per segment, usually one to two sentences. Similar to APP-350, our data are collected from Android mobile applications, and our segmentation algorithm generates policy segments with their length similar to that of APP-350.

One of the existing issues in the classification of privacy policies is the lack of negative samples. As we can see from the data distribution of OPP-115, the positive/negative ratio is highly imbalanced. In fact, the average Negative Sample Ratio (NSR) is only $2.0\%$. Research shows that such training data in many machine learning models can result in poor performance [15]. While APP-350 addresses the class imbalance issue by introducing synthetic negative data with manual modifications, which may not be the ideal solution, its NSR is only $18.6\%$. In our proposed model, the active learner is able to select the most representative segments and balance the training set automatically.

We conduct data similarity tests on OPP-115, APP-350 and our Crowdsourcing Privacy Policy Segments (CPPS) for two reasons: to evaluate our proposed segmentation algorithm and to show that CPPS is similar enough to APP-350 that it is feasibly to use labels in the APP-350 to evaluate our model trained in CPPS.

We extract segments from the three datasets, and select $100$ in each category: contact, location, and device. We evaluate similarity on segments rather than the entire privacy policies because our classifiers are trained on policy segments. Note that even within a set of data obtained from a single source, the individual policies may be very different from one another in terms of the way they are structured, the length of the policies, the complexity, and the wording preference. To address this, we introduce three similarity measurements:

1. 1.

   Intra-comparison (Intra): Evaluate data similarity within the same corpus, compare similarity of segments that share the same labeling categories.

2. 2.

   Inter-comparison (Inter): Evaluate data similarity across different corpora on the same categories. (Eg., location labels in APP-350 vs. location labels in CPPS)

3. 3.

   Inter-comparison on re-segmented policies (Re-inter): Instead of using the original policy segments in OPP-115 and APP-350, which were extracted manually by human lablers, we re-segment the complete policy texts of these two datasets using Calpric’s segmentation algorithm described in Section IV-B, and evaluate inter-similarity on these segments.

While many existing similarity measurements are intolerant of disjoint distributions, WMD is able to produce a smooth measure even if two sentences do not share any word in common. We therefore select WMD and use it on top of our customized Fasttext word embeddings and apply it to the 100 segments from each dataset. The overall intra-similarity for a corpus is defined as the average of all text similarity measured across the selected segments. The similarity value is represents the distance between sentences. In other words, a lower distance value indicates a higher similarity. We denote this as Similarity Distance (SD).

To evaluate our segmentation tool, we tabulate Inter/Re-inter in Table V by the respective values in each cell separated by slashes. Re-inter similarity distance are generally lower than Inter similarity distances due to the original segments in the OPP-115 and APP-350 being created differently. However, the average difference between Inter and Re-inter is reasonably small, only $0.3\%$. We therefore conclude that our automatic segmentation method produces similar segments as the ones manually created by skilled human labelers.

Our following experiments use a testing set of segments drawn from the APP-350 dataset. To ensure that the underlying distribution of these datasets are similar, we compare Inter and Intra across all three datasets. The three corpora share similar Intra, resulting in an average of $0.75$. Compared to this value, Inter across different corpora indeed has a slightly higher value, with an average SD of $0.78$, indicating a slightly larger difference across different corpora than within the same set. On closer inspection, we observe that APP-350 and CPPS are very similar in terms of segment structure and labels ($0.77$), whereas OPP-115 and CPPS is the most distinct combination among all ($0.80$). We therefore conclude that it is feasible to use APP-350 as the validation set, since the inter-similarity between APP-350 and CPPS is relatively close to the intra-similarity of CPPS.

In this section, we demonstrate that the proposed PPWE-biLSTM gives an accuracy similar to the state-of-the-art BERT model, while significantly outperforming the baseline LR classifier, which was widely used in previous privacy policy studies.

We randomly selected $450$ segments from each category of the unlabeled training pool $\mathbb{X}^{U}$. The segments were labeled by the crowdsourcing workers. We applied our post-processing algorithms to discard low confidence labels, and consolidated them into three sets of training data for contact, location, and device, each containing $300$ labeled segments. We prepared three test sets, each containing $300$ labeled segments randomly selected from the APP-350 Corpus.

For each classification algorithm, we trained three classifiers and evaluate them using the prepared test sets. The experimental results are shown in Table VI. We use biLSTM as an abbreviation for PPWE-biLSTM. We use two metrics to measure accuracy: F1 score and Matthew Correlation Coefficient (MCC). MCC, which is equivalent to the mean square contingency coefficient, represents the correlation between target and predictions. We include the formula in Appendix B. In binary classification, MCC is more informative since it takes into account the balance ratios of the four confusion matrix categories, especially when class imbalance issues [4] exist. The value varies between $-1$ and $+1$, where $1$ is a perfect agreement between the actual and predicted labels, $-1$ is a perfect disagreement, and $0$ occurs when the predictions are random with respect to the actuals. We include this additional metric since accuracy and F1 scores are asymmetric and sensitive to data imbalance. We perform an additional experiment using the OPP-115 and APP-350 Corpora. Similar to the previous experiment, the classifiers are trained on $300$ labels selected from each of the three categories in the datasets. As an example, Table VII shows the results for the location classifiers. Results for other categories are included in Appendix F.

From the two tables we observe:

1. 1.

   Both PPWE-biLSTM and BERT outperform the LR baseline model significantly, with an average F1 score of $89.7\%$ and $90.1\%$ versus $67.4\%$, and an average MCC value of $61.3\%$ and $63.8\%$ versus $10.7\%$.

2. 2.

   Based on the F1 score and MCC values, our PPWE-biLSTM model achieves an accuracy similar to BERT, whereas its training time is significantly shorter than that of BERT. We therefore proceed to investigate active learning approaches with the PPWE-biLSTM model.

3. 3.

   The average training accuracy, F1 and MCC of PPWE-biLSTM are $98.2\pm 1.2\%$, $98.4\pm 0.9\%$, and $97.1\pm 1.6\%$ respectively. Compared with the testing results, there is a reasonable difference of $<10\%$. As an additional example, Figure 2 shows the trend of F1 and MCC values for contact classifiers trained on $2000$ labeled segments, further confirms that these setups do not suffer from over-fitting.

4. 4.

   Although the classifiers trained on OPP-115 have very high accuracy, precision and recall values, the MCC accuracy is inferior due to the unbalanced dataset. The lack of negative samples results in nearly no training on DoesNot data actions. That is, the classifier predicts every label as positive.

Moving to the performance evaluation of active learning, the following experimental setups are used in all tests related to this topic. We specify BATCH_SIZE=$20$ for the initial boot-strap phase which is non-AL, and BATCH_SIZE=$8$ for the stage two active learning phase even though the number of new instances labeled by crowdsourcers may not be a constant value. A large training pool leads to a significant amount of memory consumption during training. For testing efficiency, we limit the unlabeled training pool $\mathbb{X}^{U}$ to contain $12K$ segments, approximately $4K$ for each category, randomly selected from the 52K policies crawled from Google Play. Note that the actual labeled training set contains fewer segments, as some instances queried by the learner do not pass the AT and fail to be aligned. Unless specified otherwise, we repeat each experiment three times and calculate the average accuracy.

We conduct two sets of experiments using different validation data. Both test sets are generated from the APP-350 corpus, and consist of $300$ labels for each category. The difference is that labels in Test_set_1 are randomly selected while Test_set_2 is selected to be a balanced set. The former is used to evaluate the model accuracy against the APP-350 dataset, whereas the latter represents a generalized true classification accuracy.

For the following experiments, we boot-strap our classifiers with $100$ labeled segments. The initial data are prepared by random selection from the unlabeled pool $\mathbb{X}^{U}$, and are labeled and consolidated using the rule of majority. The labeled training data that are ready to be used are denoted as $\mathbb{X}^{L}$. In each active learning iteration, the active learner queries new instances from $\mathbb{X}^{U}$. The selected instances are labeled by Turkers and the aligned labels will be added to $\mathbb{X}^{L}$. Note that the percentage of negative samples in the initial $\mathbb{X}^{L}$ differ from one category to another: contact has the highest negative sample rate, 31%, whereas the percentage is 23% for location, and only 8% for device.

In addition to the four querying strategies mentioned in Section VI-B1, we also include a baseline non-AL model BASE for comparison, which uses random sampling to obtain new instances. In both sets of experiments, we measure the performance of querying algorithms using F1 score and MCC, and conduct the same experiments for all three categories. We note that EER does suffer from a much slower training speed compared to other methods, as it requires a complete walk-through of all instances to calculate the expected error for each active learning iteration. However, the classifier training time is still overshadowed by the wait time for crowdsourcing tasks, which could take days to complete. However, we point out that if we continued our experiments and labeled more segments, the training time would continue to grow while the labeling time will stay constant.

Figure 3 and Figure 4 compare the F1 and the MCC performance for different querying strategies in location classifiers evaluated on Test_set_1. In both figures, each querying algorithm is associated with individual scores presented by dots, and an estimated trendline based on its average performance scores. We observe that all active learning models outperform the baseline. As a reference, while the active models converge faster, BASE reaches an F1 of $98.3\%$ with more than $2200$ labels, denoted as the converged F1 value. Figure 3 shows that, as the most effective active learning model, BMU reaches an F1 of $93.4\%$($95\%$ of the converged F1 value) with $191$ labels while BASE needs $375$ for the same score. That is, the best active learning model uses $50.93\%$ of the training labels used by BASE to achieve the same result. Similarly, BMU is able to achieve $F1=97.3\%$ ($99\%$ of the converged F1 value) using $749$ labels, which is only $48.97\%$ of the number of training samples used in BASE.

To evaluate how much saving in training labels there is when using active learning, we introduce two Percentile Scores (PS): PS_high and PS_low, representing the 90th percentile and 85th percentile for MCC, and the 99th percentile and 95th percentile for F1 score, since F1 values converge faster. We also define Training Effort Percentage (TEP) for each Percentile Score (PS) as $TEP=\frac{n_{AL}}{n_{BASE}}$, where $n_{BASE}$ is the number of training samples used in BASE to achieve a specific performance goal, and $n_{AL}$ is that number of the best case active learning model. A lower score value indicates a more effective active learning algorithm. That is, a querying strategy uses fewer training samples to achieve the same performance.

Going back to the example of Figure 3, for location classifiers, the $48.97\%$ we calculated is the TEP of $PS_{low}$ for F1 score, whereas its $PS_{low}$ is $50.93\%$, which are also shown in Table VIII. In general, while all active learning querying models outperform BASE, BMU has the best performance among all. We also observe that, compared to the baseline, the largest amount of training effort saved when using active learning models occurs in the device classifiers, where the initial training set is the most imbalanced. The improvements in contact classifiers are relatively small compared to the other two categories, as the contact dataset contains more negative samples. Further evaluations on active learning models and class imbalance will be presented in Section VII-G.

We also make an observation from Table VIII that the most effective active learning models are the ones trained on the device category, whereas the contact active learning classifiers do not save as much labeling effort. This is reasonable since privacy policy segments on device information tend to be simpler and more straightforward than contact and location. The average length of device segments is also slightly shorter than the other two. As an example, indications for device labels are mainly “device information” and “IP address”, whereas the contact category contains a lot more relevant keywords, such as email, phone number, contact book, different social network accounts such as Facebook, Twitter, or Google. Moreover, the keyword “contact” may also refer to other meanings. For instance, most privacy policies include a section for users to contact the App developers. Such sentences are also associated with email address or phone number, making it difficult for the classifiers to differentiate such segments from the actual data practice of collection/usage of contact information. As a result, it requires more labeling effort to achieve a nearly converged accuracy.

In the second set of experiments, we evaluate the models using the balanced validation set Test_set_2. As the initial training set has significant class imbalance issues, performance scores on Test_set_2 converge slower than that of Test_set_1. We therefore focus on the early stage of training and investigate the amount of labeling effort saved by active learning models to achieve an early performance goal. We set the stopping criteria for all classification models to be: $MCC>0.2$ and $F1>70\%$. We run experiments on the three categories, and observe similar results as those for Test_set_1: All active learning models outperform BASE, with BMU being the faster model to reach the stopping goal. Figure 5 summarizes the difference in labeling effort when using active learning and non-AL BASE models. We conclude that our active learning model is able to achieve the same performance with fewer training labels, whether the test sets are perfectly balanced or reflect the actual data distribution of the input space.

Similar to the previous experiments, the $TE$ percentage is calculated using the ratio of ${n_{AL}}$ and ${n_{BASE}}$. However, we observe a difference in these TE values: the device active learning classifier has a higher TE than the other two categories, while the TE for contact classifier drops to the lowest. This is also reasonable, as the second experiment is done in the early stage of a training process. Looking at the actual number of training samples shown in Figure 5, we observe that classifiers for all data categories require approximately $200$ labels to be able to surpass a $70\%$ F1 score, and $280$ labels to achieve an $MCC$ higher than $0.2$.

To investigate the relationship between AT values and the learning speed, we experiment on three different AT values: $100\%$, $80\%$, and $60\%$ using the same active learning model. The testing model is implemented with BMU, which was shown to be the best querying strategy in the previous section. We aim to introduce $30$ new labels in each active learning iteration, therefore we set the learner to query $42$ unlabeled segments per iteration, according to the average alignment rate of $73\%$. After the annotation, Calpric downloads the resulting labels and process them with the rule of majority. It consolidates these labels only when the number of majority votes reaches the testing threshold. For instance, if the current testing $AT=1.0$, we train our classifiers using only the perfectly aligned labels (all workers agree on the same label), and discard the remaining ones where AR $<$ AT.

We define each crowdsourcing label obtained from Amazon mTurk as one unit of labeling effort (LE). In other words, LE takes into account the segments sent for labeling but do not reach the minimum AT. In this experiment, we allocate $8000$ units of LE to each AT value during the active querying phase. That is, the active learning model is able to query up to $1600$ different unlabeled segments, each labeled by five Turkers, even if some of them will not align. We set the boot-strap LE at 100 for each classifier, which means they will probably end up with different numbers of initial training labels because of the different AT values. We train classifiers separately using the post-processed aligned labels according to each pre-set AT, and evaluate their performance using the balanced validation set: Test_set_2.

Figure 6 shows the trend of learning speed vs. MCC performance. Both $AT=0.8$ and $AT=1.0$ outperform $AT=0.6$. While they share a similar performance trend, $AT=1.0$ is less stable, as shown by the large fluctuations, as its training set is significantly smaller than that of $AT=0.8$ and $AT=0.6$. We include a figure showing the relationship between the number of aligned labels and the amount of LE allocated in Appendix E. On the other hand, despite having a large training set, $AT=0.6$ results in the worst performance. This is because the low AT value allows labels with low confidence to be added to the dataset, resulting in unwanted label noise. We include the graph for F1 improvement trend in Appendix D as it is not as obvious as that of MCC. In general, the use of active learning results in a larger improvement in MCC performance compared to the F1 score, because active learning in our study naturally targeted the class imbalance problem, and MCC is a direct measurement of how the classifiers are making balanced predictions. In practice, to improve the MCC performance, one should train the model using a more balanced dataset. On the other hand, the most effective way to increase the F1 score is to train models using a larger training set. Our AT experiments are done on relatively small training sets with an average size of $847$ aligned labels. As a result, the improvement in F1 will be rather insignificant.

The full performance comparison is summarized in Table IX. Although in most cases $AT=1.0$ and $AT=0.8$ outperform $AT=0.6$, we do observe a difference in the results for the location classifiers only: When AT is set to $1.0$, the both F1 and MCC performance drop to the worst among all three AT values.

While other classifiers are trained on samples with similar NSR despite being set to different AT, the location classifiers are not. We calculated the standard deviation for NSR in the three categories: $0.0425$ for contact, $0.0301$ for device, and $0.1239$ for location. Specifically, labels consolidated using $AT=0.6$ and $AT=0.8$ contain an average of $52.0\%$ and $52.1\%$ negative samples, respectively, whereas $AT=1.0$ only has an NSR of $25.5\%$. Furthermore, as the AT value increases, the total number of aligned labels decreases, together resulting in only $138$ negative labels for the $1.0$ classifier. It is possible that the some crowdsourcers did not agree on labels of potential negative samples. Because we set AT to be $100\%$, any misalignment will result in the failure to produce to pass the AT. These unaligned labels are not allowed to be added into the training set. Since we test on a relatively small $\mathbb{X}^{U}$ (4K unlabeled segments), we may have run out of negative samples quickly, and the active learner would not able to query more informative segments to be labeled. Being trained on an imbalanced dataset, the $Location-AT=1.0$ model suffers from under-fitting, and its F1 accuracy and MCC reflect this. We explore re-labeling methods to handle discarded labels in Section VII-F, and further investigate the impact of negative samples in Section VII-G.

Also note that the AR for $AT=0.8$ obtained in this experiment seems to be lower than that in previous experiments ($73\%$) in which instances are randomly selected. We suspect this is due to the active selection bias, where the queried labels contain more uncertainty than others.

From the above experiments, we conclude that $AT=0.8$ is the appropriate pre-set threshold for active learning policy classification systems of a size similar to that of our prototype tool. That being said, given a very large unlabeled training pool and an unlimited amount of LE, $AT=1.0$ does achieve the performance ceiling, as it has the lowest possibility of introducing additional label noise to the training set.

In theory, we should not discard unaligned labels in active learning classification, as they do contain a significant amount of information, which could potentially improve the model performance. As described in Section VI-B3, we implement two types of methods to handle unaligned labels: Label and Discard (L&D) and Incremental Re-Labeling (IRL). In this section, we compare these strategies using the following setup: active learning models with BMU querying strategy and AT value set to $80\%$. We use the $12K$ training pool and evaluate the classifiers using Test_set_2.

We run the IRL model on classifiers based on the three different categories, and allow each classifier to re-label up to $100$ segments. The number is denoted as the allocated re-labeling resource. As mentioned, we set the maximum number of labeling iterations to be $N=3$. Under such circumstances, if we re-label one segment ($N=2$) and a total of $10$ crowdsourcers still could not agree on a single label, we can once again publish this segment to be labeled ($N=3$), resulting in a total of $15$ labels. That being said, if we publish the same segment twice, we count this as two units used in the re-labeling resource.

We introduce a measurement of how effective the re-labeling process is. The re-labeling Success Rate (RSR) is defined as a ratio of the number of successfully aligned labels after re-labeling and the total allocated re-labeling resource. We calculate RSR for each classifier. Unfortunately, the results for relabeling are not promising. Out of 100 relabeled segments, only 2 passed the AT after $N=2$ and only 1 passed the AT after $N=3$, resulting in an overall RSR of $3\%$.

One possible reason for such a low RSR is that, because our question surveys are carefully designed and reviewed, with the additional qualification tests, crowdsourcers are able to provide correct labels as long as they pay attention to the questions. Furthermore, if the segment is longer and more complex than usual, the possibility of misinterpretation increases, affecting the successful rate of label alignment. This is proven by our experiment, as most re-labeled and aligned labels have longer lengths than the average.

Another consideration is that, some labels are ambiguous by nature. For instance, examples of the relabeled and failed include: “your GPS geo-location is not accessed without your consent” and “you will be asked for your permission each time a location-based service is requested to provide you with local search results.” It is obvious that the software is trying to access user information in order to perform a certain service. However, users also have the choice to disable such functionalities, or opt out of specific features. Such segments, even when crowdsourcers agree on the label itself, may not reflect the actual legal content. What is worse, these labels may introduce additional data noise and contaminate the training set.

We conclude that developing a more complex re-labeling design may not be an ideal choice to avoid under-fitting. Many of the segments that do not reach AT the first time are likely ambiguous and are unlikely to pass AT if more labels are obtained from Turkers, and thus it is generally a more efficient use of resources to discard such segments. Instead, we should construct a larger unlabeled training pool $\mathbb{X}^{U}$ for the active learning system to query as many informative labels as possible.

In previous experiments, we observe that the active learning models not only improve the training speed, but also solve the existing problem of class imbalance in privacy policy classification.

Figure 7 shows the trends of NSR for the labeled training set $\mathbb{X}^{L}$ using different querying strategies. We observe steep rises in LC, BMU, and EER, and a slightly higher NSR in ID than that of BASE. This is reasonable, as the initial boot-strap training set contains fewer negative samples. The classifiers are more uncertain on such instances, and query more of them during each active learning iteration. We note that all NSR converge back to the initial value as the size of $\mathbb{X}^{L}$ increases. This is due to our limited $\mathbb{X}^{U}$, which contains only $4K$ unlabeled segments that can be potentially queried. As mentioned, the number of negative samples in privacy policies tend to be much fewer than that of the positive ones. Thus, it is likely that all negative samples are used up as the training process continues.

Figure 8 confirms this hypothesis. All active learning models query more negative instances than the random baseline model. As a result, while there are still remaining negative samples in $\mathbb{X}^{U}$ for BASE, other models run out of negative samples in the first few hundreds of learning iterations. For future work, it will be crucial to obtain a reasonably large $\mathbb{X}^{U}$ before training the active learning models to ensure the set contains an adequate number of negative samples. We can approximate the amount of negative training labels needed to achieve certain goals, and back calculate the required size of unlabeled training pool $\mathbb{X}^{U}$ using the estimate percentage of NSR in the pool.