Towards Unification of Discourse Annotation Frameworks

A text is not a simple collection of isolated sentences. These sentences generally appear in a certain order and are connected with each other through logical or semantic means to form a coherent whole. In recent years, modelling beyond the sentence level is attracting more attention, and different natural language processing (NLP) tasks use discourse-aware models to obtain better performance, such as sentiment analysis (Bhatia et al., 2015), automatic essay scoring (Nadeem et al., 2019), machine translation (Sim Smith, 2017), text summarization (Xu et al., 2020) and so on.

As discourse information typically involves the interaction of different levels of linguistic phenomena, including syntax, semantics, pragmatics and information structure, it is difficult to represent and annotate. Different discourse theories and discourse annotation frameworks have been proposed. Accordingly, discourse corpora annotated under different frameworks show considerable variation, and a corpus can be hardly used together with another corpus for natural language processing (NLP) tasks or discourse analysis in linguistics. Discourse parsing is a task of uncovering the underlying structure of text organization, and deep-learning based approaches are used in recent years. However, discourse annotation takes the whole document as the basic unit and is a laborious task. To boost the performance of neural models, we typically need a large amount of data.

Due to the above issues, the unification of discourse annotation frameworks has been a topic of discussion for a long time. Researchers have proposed varied methods to unify discourse relations and debated over whether trees are a good representation of discourse (Egg and Redeker, 2010; Lee et al., 2008; Wolf and Gibson, 2005). However, existing research either focuses on mapping or unifying discourse relations of different frameworks (Bunt and Prasad, 2016; Benamara and Taboada, 2015; Sanders et al., 2018; Demberg et al., 2019), or on finding a common discourse structure (Yi et al., 2021), without giving sufficient attention to the issue of relation mapping. There is still no comprehensive approach that considers unifying both discourse structure and discourse relations.

Another approach to tackling the task is to use multi-task learning so that information from a discourse corpus annotated under one framework can be used to solve a task in another framework, thus achieving synergy between different frameworks. However, existing studies adopting this method (Liu et al., 2016; Braud et al., 2016) do not show significant performance gain by incorporating a part of discourse information from a corpus annotated under a different framework. How to leverage discourse information from different frameworks remains a challenge.

Discourse information may be used in down-stream tasks. Huang and Kurohashi (2021) and Xu et al. (2020) use both coreference relations and discourse relations for text summarization with graph neural networks (GNNs). The ablation study by Huang and Kurohashi (2021) shows that using coreference relations only brings little performance improvement but incorporating discourse relations achieves the highest performance gain. While different kinds of discourse information can be used, how to encode different types of discourse information to improve discourse-awareness of neural models is a topic that merits further investigation.

The above challenges motivate our research on unifying different discourse annotation frameworks. We will focus on the following research questions:

RQ1: Which structure can be used to represent discourse in the unified framework?

RQ2: What properties of different frameworks should be kept and what properties should be ignored in the unification?

RQ3: How can entity-based models and lexical-based models be incorporated into the unified framework?

RQ4: How can the unified framework be evaluated?

The first three questions are closely related to each other. Automatic means will be used, although we do not preclude semi-automatic means, as exemplified by Yi et al. (2021). We will start with the methods suggested by existing research and focus on the challenges of incorporating different kinds of discourse information in multi-task learning and graphical models.

The unified framework can be used for the following purposes:

1. 1.

   A corpus annotated under one framework can be used jointly with another corpus annotated under a different framework to augment data, for developing discourse parsing models or for discourse analysis. We can train a discourse parser on a corpus annotated under one framework and compare its performance with the case when it is trained on augmented data, similar to Yi et al. (2021).

2. 2.

   Each framework has its own theoretical foundation and focus. A unified framework may have the potential of combining the strengths of different frameworks. Experiments can be done with multi-task learning so that discourse parsing tasks of different frameworks can be solved jointly. We can also investigate how to enable GNNs to better capture different kinds of discourse information.

3. 3.

   A unified framework may provide a common ground for exploring the relations of different frameworks and validating annotation consistency of a corpus. We can perform comparative corpus analysis and obtain new understanding of how information expressed in one framework is conveyed in another framework, thus validating corpus annotation consistency and finding some clues for solving problems in a framework with signals from another framework, similar to Poláková et al. (2017) and Bourgonje and Zolotarenko (2019).

RST-DT is annotated on texts from the Penn Treebank (Marcus et al., 1993) that have also been annotated in PDTB. The texts are formally written Wall Street Journal articles. The English corpora annotated under the SDRT framework, i.e., the STAC corpus (Asher et al., 2016) and the Molweni corpus (Li et al., 2020), are created for analyzing multi-party dialogues, making it difficult to be used together with the other two corpora. Therefore, in addition to RST-DT and PDTB 3.0, we will use the ANNODIS corpus (Péry-Woodley et al., 2009), which consists of formally written French texts. We will first translate the texts into English with an MT system and then manually check the translated texts to reduce errors.

In the following, the research questions and the approach in our plan will be discussed. These questions are closely related to each other and the research on one question is likely to influence how the other questions should be addressed. They are presented separately just for easier description.

RQ1: Which structure can be used to represent discourse in the unified framework?

Although there is a lack of consensus on how to represent discourse structure, in a number of studies, the dependency structure is taken as a common structure that the other structures can be converted to (Muller et al., 2012; Hirao et al., 2013; Venant et al., 2013; Li et al., 2014; Yoshida et al., 2014; Stede et al., 2016; Morey et al., 2018; Yi et al., 2021). This choice is mainly inspired by research in the field of syntax, where the dependency grammar is better studied and its computational and representational properties are well-understood¹¹1In communication with Bonnie Webber, January, 2022.. The research by Venant et al. (2013) provides a common language for comparing discourse structures of different formalisms, which is used in the transformation procedure presented by Stede et al. (2016). Another possibility is the constrained directed acyclic graph introduced by Danlos (2004). While Venant et al. (2013) focus on the expressivity of different structures, the constrained DAG is motivated from the perspective of strong generative capacity (Danlos, 2008). Although neither of the studies deals with the PDTB framework, since they are both semantically driven, we believe it is possible to deal with the PDTB framework using either of the two structures. We will start with the investigation of the two structures.

Another issue is how to maintain one-to-one correspondence during the transformation of the original structure and the unified structure back and forth. As indicated by Stede et al. (2016), the transformation from the RST or SDRT structures into dependency structures always produces the same structure, but going back to the initial RST or SDRT structures is ambiguous. Morey et al. (2018) introduces head-ordered dependency trees in syntactic parsing (Fernández-González and Martins, 2015) to reduce the ambiguity. We may start with a similar method.

As is clear from Section 2, using the dependency structure as a common ground for studying the relations between different frameworks is not new in existing literature, but comparing the RST, PDTB and SDRT frameworks with this method has not yet been done. This approach will be our starting point, and the suitability of the dependency structure in representing discourse will be investigated empirically. The SciDTB corpus (Yang and Li, 2018), which is annotated under the dependency framework, will be used for this purpose.

RQ2:²²2In communication with Bonnie Webber, January, 2022. We thank her for pointing out this aspect. What properties of different frameworks should be kept and what properties should be ignored in the unification?

We present a non-exhaustive list of properties, which we consider to have considerable influence on the unified discourse structure.

1. 1.

   Nuclearity: Marcu (1996) uses the nuclearity principle as the foundation for a formal treatment of compositionality in RST, which means that two adjacent spans can be joined into a larger span by a rhetorical relation if and only if the relation holds between the most salient units of those spans. This assumption is criticized by Stede (2008). The remedy provided by Stede (2008) is to separate different levels of discourse information, which is in line with the suggestions in Knott et al. (2000) and Moore and Pollack (1992). Our strategy is to keep this property in the initial stage of experimentation. The existing methods for transforming RST trees to dependency structure (Hirao et al., 2013; Li et al., 2014) rely heavily on the nuclearity principle and we will use these methods in the transformation and see what kinds of problems this procedure will cause, particularly with respect to the PDTB framework, which does not enforce a hierarchical structure for complete coverage of the text.

2. 2.

   Sentence-boundedness: The RST framework does not enforce well-formed discourse sub-trees for each sentence. However, it is found that 95% of the discourse parse trees in RST-DT have well-formed sub-trees at the sentence level (Soricut and Marcu, 2003). For the PDTB framework, there is no restriction on how far an argument can be from its corresponding connective: it can be in the same sentence as the connective, in the sentence immediately preceding that of the connective, or in some non-adjacent sentence (Prasad et al., 2006). Moreover, the arguments are determined based on the Minimality Principle, which means that clauses and/or sentences that are minimally required for the interpretation of the relation should be included in the argument, and other spans that are relevant but not necessary can be annotated as supplementary information, which is labeled depending on which argument it is supplementary to (Prasad et al., 2008). The SDRT framework developed in Asher and Lascarides (2003) does not specify the basic discourse unit, but in the annotation of the ANNODIS corpus, EDU segmentation follows similar principles as RST-DT. The formation of CDU and the attachment of relations are where SDRT differs significantly from RST. A segment can be attached to another segment from the same sentence, the same paragraph or a larger context, and by one or possibly more relations. A CDU can be of any size and can have segments that are far apart in the text, and relations may be annotated within the CDU³³3See section 3 of the ANNODIS annotation manual, available through http://w3.erss.univ-tlse2.fr/textes/publications/CarnetsGrammaire/carnGram21.pdf.

   The differences in the criteria on location and extent for basic discourse unit identification and relation labeling of the RST framework and the PDTB framework may be partly attributed to different annotation procedures. In RST, EDU segmentation is performed first and EDU linking and relation labelling are performed later. The balance between consistency and granularity is the major concern behind the strategy for EDU segmentation (Carlson et al., 2001). In contrast, in PDTB, the connectives are identified first, and their arguments are determined afterwards. Semantic relatedness is given greater weight and the location and extent of the arguments can be determined more flexibly. On the whole, neither SDRT nor PDTB shows any tendency of sentence-boundedness. We will investigate to what extent the tendency of sentence-boundedness complicates the unification and what the consequences are if entity-based models and lexical-based models are incorporated.

3. 3.

   Multi-sense annotation: As shown above, SDRT and PDTB allow multi-sense annotation while RST only allows one relation to be labeled. The single-sense constraint actually gives rise to ambiguity because of the multi-faceted nature of local coherence (Stede, 2008). For the unification task, we assume that multi-sense annotation is useful. However, we agree with the view mentioned in Stede (2008) that incrementally adding more relations as phenomena are being recognized is not a promising direction. There are two possible approaches: one is to separate different dimensions of discourse information (Stede, 2008) and the other is to represent different kinds of discourse information simultaneously, similar to the approach adopted in Knott et al. (2000). While multi-level annotation may reveal the interaction between discourse and other linguistic phenomena, it is less helpful for developing a discourse parser and requires more efforts in annotation. The second approach may be conducive to computationally cheaper discourse processing when proper constraints are introduced.

RQ3: How can entity-based models and lexical-based models be incorporated into the unified framework?

The PDTB framework believes that lexical-based discourse relations are associated with anaphoric dependency, which is anchored by discourse adverbials (Webber et al., 2003) and annotated as a type of explicit relations. As for entity-based relations, PDTB uses the EntRel label to annotate this type of relations when neither explicit nor implicit relations can be identified and only entity-based coherence relations are present. In the RST framework, the ELABORATION relation is actually a relation between entities. However, it is encoded in the same way as the other relations between propositions, which bedevils the framework (Knott et al., 2000). Further empirical studies may be needed to identify how different frameworks represent these different kinds of discourse information. The main challenge is to use a relatively simple structure to represent different types of discourse information while keeping the complexity relatively low.

RQ4: How can the unified framework be evaluated?

We will use intrinsic evaluation to assess the complexity of the discourse structure.

Extrinsic evaluation will be used to assess the effectiveness of the unified framework. The downstream tasks in the extrinsic evaluation include text summarization and document discrimination, which are two typical tasks for evaluating discourse models. The document discrimination task asks a score of coherence to be assigned to a document. The originally written document is considered to be the most coherent, and with more permutations, the document becomes less coherent. For comparison with previous studies, we will use the CNN and Dailymail dataset (Hermann et al., 2015) for the text summarization task, and use the method and dataset⁴⁴4https://github.com/AiliAili/Coherence_Modelling in Shen et al. (2021) to control the degree of coherence for the document discrimination task.

Previous studies that use multi-task learning and GNNs to encode different types of discourse information will be re-investigated to test the effectiveness of the unified framework.

As we may have to ignore some properties, we will examine what might be lost with the unified framework.

We propose to unify the RST, PDTB and SDRT frameworks, which may enable discourse corpora annotated under different frameworks to be used jointly and achieve the potential synergy of different frameworks. The major challenges include determining which structure to use in the unified framework, choosing what properties to keep and what to ignore, and incorporating entity-based models and lexical-based models into the unified framework. We will start with existing research and try to find a computationally less expensive way for the task. Extensive experiments will be conducted to investigate how effective the unified framework is and how it can be used. An empirical evaluation of what might be lost through the unification will be performed.

We thank Bonnie Webber for valuable feedback that greatly shaped the work. We are grateful to the anonymous reviewers for detailed and insightful comments that improved the work considerably, and Mark-Jan Nederhof for proof-reading the manuscript. The author is funded by University of St Andrews-China Scholarship Council joint scholarship (NO.202008300012).

The corpora are used in compliance with the licence requirements:

The ANNODIS corpus is available under Creative Commons By-NC-SA 3.0.

RST-DT is distributed on Linguistic Data Consortium:

Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. RST Discourse Treebank LDC2002T07. Web Download. Philadelphia: Linguistic Data Consortium, 2002.

PDTB 3.0 is also distributed on Linguistic Data Consortium:

Prasad, Rashmi, et al. Penn Discourse Treebank Version 3.0 LDC2019T05. Web Download. Philadelphia: Linguistic Data Consortium, 2019.

Bender Rule English is the language studied in this work.