LEAN-LIFE: A Label-Efficient Annotation Framework
Towards Learning from Explanation

Deep neural networks have achieved state-of-the-art performance on a wide range of sequence labeling and classification tasks such as named entity recognition (NER) Lample et al. (2016); Ma and Hovy (2016), relation extraction (RE) Zeng et al. (2015); Zhang et al. (2017); Ye et al. (2019), and sentiment analysis (SA) Wang et al. (2016). However, they only yield such performance levels in supervised learning scenarios, and in particular when human-annotated data is abundant. As we seek to apply NLP models to larger variety of domains, such as product reviews Luo et al. (2018), social media messages Lin et al. (2017), while reducing human annotation efforts, better annotation frameworks with label-efficient learning techniques are crucial to our progress.

Annotation frameworks have been explored by several previous works Stenetorp et al. (2012); Bontcheva et al. (2014); Morton and LaCivita (2003); de Castilho et al. (2016); Yang et al. (2017). These existing open-source sequence annotation tools mainly focus on optimizing user-friendly user interfaces, such as providing shortcut key functionality to allow for faster tagging. The frameworks also attempt to provide annotation recommendation to reduce human annotation efforts. However, these recommendations are provided by a pre-trained model or via dictionary look-ups. This methodology of providing recommendations often proves to be unhelpful when little annotated data exists for pre-training, as is usually the case for natural language tasks being applied to domain-specific or user-provided corpora.

To resolve this issue, AlpacaTag, an annotation framework for sequence labeling  Lin et al. (2019) attempts to provide annotation recommendations from a learned sequence labeling model that is incrementally updated by batches of incoming human annotations. Its model training follows an active learning strategy Shen et al. (2017), which is shown to be a label-efficient, thus it attempts to minimize human annotation efforts. AlpacaTag selects the most informative batches of documents for humans to annotate and thus achieves a more cost-effective way of using human efforts. While active learning allows the model to achieve higher performance earlier in the learning process, model performance could be improved if additional supervision existed. It is imperative that provided annotation recommendations be as accurate as possible, as inaccurate annotation recommendations from the framework can push users towards generating noisy data, hindering instead of aiding the model training process.

Our effort to prevent this problem is centered around allowing annotators to provide additional supervision by capturing labeling explanations, while still taking advantage of the cost-effectiveness of active learning. Specifically, as shown in  Fig. 1, we allow annotators to provide explanations for their decisions in natural language or by selecting triggers––nearby phrases that provide helpful context for their decisions. These enhanced annotations allow for model training over both user-provided labels, as well as weakly labeled data created by parsing explanations into high precision labeling rules. We therefore make attempts to ameliorate the erroneous recommendation problem by a performance-boosting training strategy that incorporates both labeled and unlabeled data.

Our work is also similar to recent attempts that exploit explanations for an improved training process Srivastava et al. (2017); Hancock et al. (2018); Zhou et al. (2020); Qin et al. (2020), but with two main differences. First, we embed this improved training process in a practical application and second, we design task specific architectures to incorporate the now captured explanations into training.

To the best of our knowledge, there is no existing open-source, easy-to-use, recommendation-providing, online-learning annotation framework that can also capture explanations. LEAN-LIFE is the first framework to capture and leverage explanations for improved model training and performance, while still inheriting the advantages of existing tools. We summarize our contributions as:

As shown in  Fig. 2, our framework consists of two main components, a user-friendly web-UI that can capture labels and explanations for labeling decisions, and a weak supervision framework that parses explanations for the creation of weakly labeled data. The framework then uses this weakly labeled data in conjunction with user-provided labels to train models for improved annotation recommendations. Our UI shows annotators unlabeled instances (can be sampled using active learning), along with annotation recommendations in an effort to reduce annotation costs. We use PyTorch to build our models and implement an API for communication between the web-UI and our weak supervision framework. The learned parameters of our framework are updated in an online fashion, thus improving in near real time. We will first touch on the annotation UI (§3) and then go into our weak supervision framework (§4).

The emphasis of our front-end design is to simplify the capture of both label and explanation for each labeling decision, while reducing annotation effort via accessible annotation recommendation. Our framework supports two forms of explanations, Triggers and Natural Language. A Trigger is a group of words in the sentence being annotated that aided the annotator’s labeling decision, while Natural Language is a written explanation of the labeling decision. This section presents first the UI for capturing triggers (§3.1) and then the UI for capturing natural language explanations (§3.2).

Our Weak Supervision Framework is composed of two main components, a weak labeling module that parses explanations to create labeling rules and a downstream model. The framework parses user-provided explanations to generate weakly labeled data and then trains the appropriate downstream model with this augmented training data. Our weak labeling module supports both explanation formats provided to the annotator in the UI––triggers and natural language. This section first introduces how the module utilizes triggers (§4.1) and then presents how the module deals with natural language(§4.2).

We conduct extensive experiments investigating label efficiency to prove the effectiveness of our annotation models. We found that using natural language explanations for RE and SA, and trigger explanations for NER provided the best results. For the downstream model portion of our weak supervision framework, we use common supervised method for each task: (1-RE) BLSTM+ATT Bahdanau et al. (2014) adds an attention layer onto LSTM to encode an sequence. (2-SA) ATAE-LSTM Wang et al. (2016) combines the aspect term information into both the embedding layer and attention layer to help the model concentrate on different parts of a sentence. (3-NER) BLSTM+CRF Ma and Hovy (2016) encodes character sequences into a vector and concatenates the vector with pre-trained word embeddings to feed into word-level BLSTM. Then, it applies a CRF layer to predict sequence labels. Then we compare these methods as baselines.

Leveraging natural language explanations for additional supervision has been explored by many works.  (Srivastava et al., 2017) first demonstrated the idea of using natural language explanations for weak labeling by jointly training a task-specific semantic parser and label classifier to generate weak labels. This method is limited though, as the parser is too tightly coupled to the already labeled data, thus their weak learning framework is not able to build a much larger dataset than the one it already has. To address this issue,  (Hancock et al., 2018) proposed a weak supervision framework that utilizes a more practical rule-based semantic parser. The parser constructs a logical form for an explanation that is then used as a labeling function––this resulted in a significant increase of the training set. Another effort to incorporate explanations can be found in  (Camburu et al., 2018) work to extend the Stanford Natural Language Inference dataset with natural language explanations––this extension was done for the important textual entailment recognition task. They demonstrate the usefulness of explanations as an additional training signal for learning more comprehensive sentence representations. Even earlier  (Andreas et al., 2016) explored breaking down natural language explanation into linguistic sub-structures for learning collections of neural modules which can be assembled into neural networks. Our framework is very related to the above weak supervision methods via explanation.

Another approach to weak supervision is attempting to transfer knowledge from a related source to the target domain corpus Lin and Lu (2018); Lan et al. (2020).  Ni et al. (2017) attempts to create weakly labeled NER data for a target language via an annotation projection from a comparable corpus. However, their efforts regard unlabeled words as ‘O’, and so it cannot deal with incomplete annotations––a feature an annotation framework must handle.  Shang et al. (2018) and  Yang et al. (2018) proposed using a domain-specific dictionary for matching on the unannotated target corpus. Both efforts employ Partial CRFs Liu et al. (2014) which assign all possible labels to unlabeled words and maximize the total probability. This approach addresses the incomplete annotation problem, but heavily relies on a domain-specific seed dictionary.

In this paper, we propose an open-source web-based annotation framework LEAN-LIFE that not only allows an annotator to provide the needed labels for a task, but can also capture explanation for each labeling decision. Such explanations enable a significant improvement in model training while only doubling per instance annotation time. This increase in per instance annotation time is greatly outweighed by the benefits in model training, especially in a low resource settings, as proven by our experiments. This is an important consideration for any annotation framework, as the quicker the framework is able to train annotation recommendation models to reach high performance, the sooner the user receives useful annotation recommendations, which in turn cut down on the annotation time required per instance.

Better training methods also allow us to fight the potential generation of noisy data due to inaccurate annotation recommendations. We hope that our work on LEAN-LIFE will allow for researches and practitioners alike to more easily obtain useful labeled datasets and models for the various NLP tasks they face.

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, NSF SMA 18-29268, and Snap research gift. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.