Calpric: Inclusive and Fine-grained Labeling of Privacy Policies with Crowdsourcing and Active Learning

OPP-115 [50] and APP-350 [49] are so far the most used privacy policy data sets for machine learning-based models, yet they both suffer from data imbalance. That is, the distribution of data categories collected by digital applications is not uniform, and as a result, privacy policies do not uniformly cover various categories of data. Imbalance exists across both data categories and positive/negative classes. For example, there are many more privacy policy text segments disclosing the collection of location information than health information. Similarly, the privacy policy segments are heavily skewed towards assertions (i.e. a positive label) of data collection than denials (i.e. a negative label).

To better characterize the challenge, we tabulate both the number of labeled segments in each category and the percentage of denials for the two data sets and CPPS in Table 4. We can see that data for certain categories is extremely sparse: OPP-115 contains thousands of labels on contact data but only $44$ labels on health data. Consequently, Polisis [19], which is trained on OPP-115, struggles with a low accuracy in minority classes such as health. We also see that the data sets exhibit class imbalance across categories, with only 4.19% of the samples in the OPP-115 having a denial label. APP-350 has a slightly better class balance, but this is due to the addition of synthetic denials, which they accomplished by manually changing a positive sample into a denial one. For instance, a positive sample: “Our App collects your location data” might be converted to “Our App does not collect your location data”. We note that while this approach mitigates the class imbalance to some extent, it cannot be used to improve data category imbalance. Furthermore, there is no assurance that such synthetic labels are representative of real denials in the wild. In comparison, after active learning, the CPPS training set generated by Calpric has almost perfect class balance. As we can see, Calpric is able to find and label many more samples in the 6 rarer data categories.

Previous data sets, such as the OPP-115 [42] and the APP-350 [49] data sets, are generated with trained human annotators, usually composed of law students. Even with their prior legal knowledge, it still takes considerable time to read through a privacy policy and label the specific text segments that indicate, for example, whether personal information is collected and if so, what type of information is collected. For instance, OPP-115, which contains 115 privacy policies, was labeled by $10$ law students, spending an average of $72$ minutes on each privacy policy. The speed, availability and cost of trained annotators limits the size of the data set. Recognizing this challenge, we propose a crowdsourcing solution reduces the cost of labeling by $9\times$.

One obvious question that arises with the use of crowdsourced annotators for privacy policies is whether such untrained annotators can sufficiently understand privacy policy language to produce accurate labels [43]. We note however, that since privacy policies are the primary method of obtaining consent from individuals for the collection of information, to obtain meaningful consent, they must be understandable by a representative target individual from whom the information collection would likely happen. For example, Canada’s privacy legislation, PIPEDA, states that “the consent of an individual is only valid if it is reasonable to expect that an individual to whom the organization’s activities are directed would understand” and the GDPR defines consent as “any freely given, specific, informed and unambiguous indication of the data subject’s wishes”—in both cases, the user must “understand” or be “informed.” While it is true that many privacy policy documents fall short of this idealized level of transparency, we argue that the interpretation that an average person has for a privacy policy intended for them (were they to actually read the policy) is still relevant, and is what Calpric tries to predict.

Nevertheless, one cannot simply use crowdsourced annotators and expect to achieve accurate classification, especially in combination with active learning. First, crowdsourced annotators acquired through platforms such as Amazon’s mTurk service, generally do best on short, simple tasks, such as single-sentence translation [1] and sentiment analysis [7]. Further, reducing the complexity and length of tasks is beneficial for crowdsourced annotators [23, 8]. Second, active learning methods generally assume reliable oracles that always return correct labels, which is not the case for crowdsourced annotators. Previous privacy policy studies presented several crowdsourced annotators with the entire privacy policy, asked them each to assign a label, and then attempt to increase label reliability by only accepting the label if the inter-annotator agreement is above some threshold [43]. A drawback to this is that different annotators may select different text segments to label, and the text segments themselves may not align perfectly, decreasing inter-annotator agreement.

Calpric addresses these challenges by extracting relevant text segments from a privacy policy and presenting these to the crowdsourced annotators so that all of them label the same segments. This both decreases the time they must spend on the task and increases inter-annotator agreement rate, which both decrease the cost of acquiring labels. Calpric re-requests labels for text segments that did not achieve an inter-annotator agreement above an acceptable threshold—a mechanism called relabeling. However, some segments may suffer from chronically poor agreement—that is they do not achieve sufficient agreement even after several attempts. There can be several reasons for the chronically poor agreement for a segment. First, Calpric may mistakenly extract a segment that is not relevant to the labeling task. Our crowdsourced task also has workers annotate whether a segment is relevant or not, which is then used as a label to retrain the model used to classify if a text segment contains relevant information about a data category. Second, the text segment may be relevant but inherently ambiguous—that is the text is not sufficiently clear about whether collection is occurring or not [32]. For example, in {quoting} “if you log-in using a third party social media account, we may not collect any personal or account information from that social media provider” the “may” makes it ambiguous whether information is collected or not, and users have no control over the usage of private data. In addition, using the ambiguous term “personal information” without further details makes it impossible to infer which data category the segment refers to and undermines the purpose and value of privacy policies [32]. To prevent wasting more crowdsourcer resources on such text segments, Calpric stops the relabeling attempt after a preset number of iterations, and labels such segments as ambiguous and does not use them for training. While classifying all text segments that do not achieve an acceptable level of inter-annotator agreement as ambiguous necessarily over-approximates the true set of ambiguous text segments, we show that Calpric is still able to achieve high classification accuracy despite the loss of potentially usable text segments due to this over-approximation.

Our current implementation of Calpric uses crowdsourced annotators from Amazon’s mTurk service to provide labels used for training. While mTurk captures a large portion of the mobile app user population, who are the target individuals of mobile app privacy policies in our data set, we note that mTurk cannot capture every subgroup of mobile app user. For example, children are not permitted to use the mTurk service, so our current Calpric models cannot predict how children might interpret privacy policies. Nonetheless, this is a limitation of mTurk, not Calpric—we believe that our current implementation of Calpric can be extended in a straightforward manner to predict for any subgroup of individuals so long as annotators from that subgroup can be recruited to provide labels from which Calpric can learn.

For each data category, Calpric trains a binary Category Model to identify relevant sentences: it projects each sentence into a vector representation using the PriBERT embedding mentioned in Section 3.2. These vectors are then fed into a set of binary classification models fine-tuned on top of BERT. We set the batch size to 32 and the number of epochs to 48. We use leaky ReLUs [29] and a dropout value  [39] of $0.1$ to prevent over-fitting while sustaining the weight updates to avoid vanishing gradient problems in the propagation process. We apply a softmax activation function to the model output and use cross entropy loss and ADAM optimizer [22] to update model weights.

Each Category Model is initialized using a bootstrap training set of 200 sentences to be labeled. The bootstrap set is prepared in two steps: first, Calpric randomly selects a number of privacy policies from the unlabeled pool, and crowdsourced workers are asked if the target data category is mentioned in these given documents. While this requires the annotators to read through the full documents, they do not need to give detailed annotations. The privacy policies that are annotated as mentioning a target data category will be tokenized into sentences, and $200$ sentences are randomly selected as the bootstrap set for each of the 9 Category Models. After this, active learning is applied to select more sentences for labeling. It is possible that the bootstrap set does not cover all classes and the model will fail to converge, but we did not encounter this situation in all our experiments. If such a situation arises, it would be necessary to expand the size of the bootstrap set until all classes are represented.

The Contextualizer starts with a relevant sentence selected by the Category Model and iteratively expands it into a text segment by examining the sentences immediately before and after to see if either should be added to the current segment. To do this, the Contextualizer computes the similarity of the sentences to the current text segment using the Word Mover’s Distance (WMD) [24] and compares it against a threshold calculated from the WMD mean and standard deviation of sentences within the current privacy policy [12]. This process continues until both the previous and subsequent sentences are not similar enough to be added to the current text segment.

As APP-350 and OPP-115 do not have labels for “choice” or “ambiguous” we set aside 30 labeled segments per data action in each data category for a total of 810 test segments. We then train Calpric without the test segments and tabulate Calpric’s accuracy in Table 3. To save space, we show the accuracy of each action mode, averaged across all three data actions in each data category, as well as an overall accuracy for each data category.

The evaluation shows that Calpric can achieve good accuracy across both common and rare data categories, as well as good accuracy in all the fine-grained action model labels. Specifically, it achieves an average f1 of 0.86 and an overall accuracy of 0.90, with the highest f1 in Assert (0.92) and Not Mentioned (0.91), and the lowest in Ambiguous (0.78). We suspect this is because Assert statements are usually described clearly as per the legal requirements, while Not Mentioned labels are very different from labels of the remaining 4 action modes which all have some contextualized relationship with the target data category. Therefore intuitively it is easier for Calpric to correctly predict on Not Mentioned labels. On the other hand, its accuracy in predicting Ambiguous labels is slightly lower, as there are many factors leading to ambiguity in privacy policies, e.g. long sentences, terms being too general, confusing wordings, incomplete information, and so on. However, we believe that an F1 score of 0.78 is still very promising, given the fact of the potential noise within the ambiguous labels.

As explained in Section 2.1, previous tools classify at a coarser granularity than Calpric, so we cannot directly compare the accuracy of labels produced by Calpric against theirs. As previously shown in Table 1, previous tools label privacy policy text segments into two classes, while Calpric provides 5. However, all previous tools were trained exclusively on the APP-350 and OPP-115 data sets, while Calpric is trained on the larger and balanced CPPS dataset that was produced by Calpric’s training approach. It is therefore unfair to compare Calpric against existing tools. Instead, we train baseline models with identical architectures as those of Calpric’s on data from APP-350 and OPP-115. We randomly select some segments from the the APP-350 and OPP-115 as the test set and tabulate the F1 result in Table 4. Calpric outperforms the baseline models on fine-grained classification tasks as well as the overall average across all data categories.

To examine the ability to capture useful segments from privacy policies, we compare the outputs of Calpric’s Segmenter against the text segments extracted by human annotators in APP-350. We measure the number of labeled sentences in APP-350 in the relevant data categories that are covered by (i.e. appear entirely in) some segment extracted by our Segmenter. We run our Segmenter on all the original privacy policies provided by APP-350 and find that $97\%$ of the sentences in APP-350 are covered by our Segmenter. We examine the uncovered sentences and find that the vast majority are either ambiguous or irrelevant sentences. For example, in APP-350, the following segment is labeled as collecting contact information: {quoting} “Anonymous Data AI Factory may collect and use anonymous data, via Google Analytics. We cannot identify you from this data. Such data is pooled to give us general statistics on our audience and usage of our apps.” but is not labeled as collecting contact information by our Segmenter.

We now evaluate the benefit of having annotators label text segments instead of existing approaches such as full-text or paragraph-level labeling. We randomly select $3$ privacy policies from APP-350 and conduct an experiment where mTurk annotators are asked to either label the full document, paragraphs produced by existing preprocessors, or text segments extracted by Calpric. For full document labeling (FDL), we present the full document to annotators and ask them to label 20 sentences that denote any of the data actions and action modes on any of the relevant data categories. For paragraph labeling (PL), we present paragraphs for annotators to select and label. We generate paragraphs using a modified version of the PolicyLint preprocessing script. We exclude outlier paragraphs falling out of the middle two quartiles (i.e. 25-75%) of the distribution in terms of length, as they might be excessively long or so short as to contain no useful information. For segment labeling (SL), we have annotators label 20 segments per privacy policy using the Calpric’s survey. To align with our labeling scheme (i.e. includes more classes), which differs from the original APP-350, selected samples are reviewed and re-labeled by law students. We calculate label accuracy (Acc.) by measuring the percentages of raw labels that are correctly labeled by mTurkers. A label is considered accurate if and only if it covers one of the labeled sentences in APP-350 and the labels in data category, data action and action mode all match.

The results are tabulated in Table 5. We found that SL produces the most efficient results: annotators needed roughly 4 times and 2 times as long to produce the same number of labels using FDL and PL, respectively. That is, 21 minutes for SL, 46.5 minutes for PL, and 90 minutes for FDL (per policy). As a result, to maintain the same hourly pay rate, FDL and PL require 4$\times$ and 2$\times$ the payment to obtain the same amount of labels as using SL. We find that SL gives the highest accuracy. Closer inspection of FDL shows that some annotators incorrectly label segments such as “We do not track your location unless we have your opt-in consent.” as “No” instead of “Choice”, as they truncated the phrase after “location”, thus missing the later part that makes it “Choice”. SL also outperforms PL by 15%. We found that this was mainly because text critical to understanding a data action may span a paragraph break.

We also observe that the agreement rate (AR) for FDL is significantly lower, as annotators often disagree on text boundaries for their labels. This result aligns with what previous scholars observed [42], even when labels are created by trained annotators. On the other hand, ARs for PL and SL are similar, with PL being slightly higher (1.3%).

We compute the raw cost per label by dividing the total payments by the number of unique labeled segments, the effective cost after discarding unaccepted segments, and the time taken to produce each raw label. FDL is both more expensive (4$\times$ for raw labels, and 5.2$\times$ for accepted labels), and requires 4.3$\times$ as much time to produce raw labels than SL. Qualitatively, we also found that because annotators are free to select the text they wish to label, they tended to select easier segments that could be found by simply searching the text for keywords (i.e. “demographic” or “contact”) instead of carefully reading the complete privacy policy. As a result, this may exacerbate class imbalance and lead to poorer coverage of the privacy policy text. PL has a better performance than FDL, yet is still 2.2$\times$ more expensive than SL.

One potential risk of segment-level labeling is that it becomes unclear how to consolidate contradictory labels if more than one relevant segment for a data category and data action appear in the same privacy policy. We examined 115 privacy policies and find that 12% have contradictory labels for a data category and data action. Of these, 8% have conflicts that take the form of a denial that they collect a certain type of data, followed by a statement that they will share that data if legally required to do so. We also compare contradictions found by Calpric with PolicyLint[2], on OPP-115. PolicyLint finds 20 of the privacy policies have contradictions, 8 of which are also found by Calpric. Of the 12 not found by Calpric, PolicyLint considers 5 contradictions only because it lacks a “Choice” label, 2 are due to PolicyLint and Calpric having different subsumptive relationships (i.e. PolicyLint considers demographic information as a type of personal information while Calpric considers them separate classes), 2 because of differences in the annotation scheme and 3 due to differences in the data categories covered. Calpric also finds 6 contradictions that PolicyLint does not find. We manually verified that they are real contradictions and were missed by PolicyLint as it is unable to differentiate between choices and explicit denials.

We evaluate the reduction in the amount of labeled training data that active learning provides to Calpric. To do this, we set up an experiment that evaluates how many training samples are required to achieve a certain Category Model accuracy across all 6 data categories. We prepare a set of balanced test and training sets for each data category by using samples from APP-350 (and OPP-115 if APP-350 does not label such categories). Text segments that are labeled as in a data category are assertions, and segments not in the category are denials. We simulate a non-active training procedure by randomly selecting samples from this pool and compare this to a training procedure that uses active learning to select samples. We record the number of training samples required to achieve an F1 score of 0.85 in the non-active learning and active learning cases in the n_nonAL and n_AL columns respectively, and the percentage of samples saved in the active learning case in the AL_save column in Table 6. We did not include survey, personal identifier or social media because the non-AL model is not able to find enough training samples to achieve the required accuracy. For the other categories, we observe that the average savings in the number of samples required is $28.52\%$. The savings are higher in categories where more samples are required in the non-active learning case, suggesting that there are many similar samples that the random selection in the non-active learning case selects over and over again. However, this trend is weaker in financial and demographic categories. Upon further investigation, we observe that these two categories exhibited longer samples with more complex features. As an example, device-related sentences such as {quoting} “Our app may access your unique device identifier to tailor ads.” are usually simple and concise, whereas a demographic-related sentence like {quoting} “App A provides us with access to certain information about such User as is stored in the User’s App A account, namely, the User’s public profile (i.e., User’s name, profile picture, email address, gender and other public information) and his/her list of friends and/or any other information which is detailed and displayed to the User in the notice which appears during the log in process.” is not. This complexity means that a good number of samples are required to achieve higher accuracy regardless of how the samples are selected, which makes the advantages of active learning in these cases, though still significant, somewhat lower than in the cases with simpler samples. The contact and device categories exhibit the greatest savings. We believe this is a result of those categories having a heavy-tailed distribution, the tails of which active learning is much better able to target than random selection.

To verify that Calpric mitigates class imbalance, we measured class balance as a percentage of samples in a minority class, recorded at the bootstrap phase and at the end of model training, respectively. Table 6 also shows the starting (m_start) and ending (m_end) percentages of minority samples in all data categories. We see that in all categories, the minority class initially starts of low, but eventually approaches close to making up half of the training set ($50\%$) used to train the model. We also observe an inverse correlation between the increase in minority percentage and the AL saving percentage. This is intuitively reasonable because boosting from a lower minority percentage to a higher minority percentage naturally requires more training samples. We performed a similar evaluation on Action Models, and found similar results in the improvement of class balance. In these cases, the increases in the minor classes were 29.6% in the health, 19.4% in the financial, and 24.0% in the demographic categories.

We also compare Calpric with standard upsampling methods to mitigate to class imbalance, such as duplicating or synthesizing minority samples [21]. To do this, we randomly select 800 labeled text segments and sort them by whether they are in a minority or majority class. We then duplicate, synthesize or select minority labels with active learning until the training sets are balanced, train our models and compare accuracy. To duplicate samples, we randomly select samples from the minority set and repeat them in the final training set. To synthesize minority samples we randomly select majority samples and negate them using the method described in [4], implemented with Stanza⁵⁵5https://stanfordnlp.github.io/stanza/. Finally, we use active learning to select enough additional minority samples to balance the training set. We then train a fine-tuned BERT model with a batch size of 32 for 48 epoches and evaluate the resulting model on a test set with a naturally occurring distribution of majority and minority samples. We repeat the experiment 5 times and tabulate the loss, accuracy, F1 values for majority and minority test sets, weighted F1, and balanced F1 for the 3 data set balancing methods, as well as training with no balancing method in Table 7. As we can see, Calpric with active learning achieves better results except for loss. Using no balancing method (None) achieves the lowest loss, but as expected, suffers from low accuracy in the minority class, as it has simply optimized for the majority class.