Revisiting Absence with Symptoms that *T* Show up Decades Later to Recover Empty Categories

Null elements, or empty categories, are elements that exist within a parse tree that do not correspond to a surface level word. Chomsky outlined how certain elements could be invisible or null in surface structure (Chomsky, 1981). These null elements serve an important role in syntax, especially in explaining movement and transformational grammars (Chomsky, 1965). In the context of dependency parsing, particularly in the context of predicate elision, null elements have predominantly surfaced in the enhanced representation of universal dependencies.¹¹1https://universaldependencies.org/u/overview/enhanced-syntax.html Such elements can encode additional semantic information of sentences by recording non-local dependencies, enabling interpretation of constructions such as WH-questions (Johnson, 2002). Conversely, within constituency parsing, null elements have typically been treated as preprocessed entities to be removed (Collins, 1999; Bikel, 2004).

However, excluding null elements can be problematic when attempting to parse more than just the syntactic information from a surface sentence, since the semantic information they provide is often lost. Certain languages, such as Japanese, Korean, and Chinese, more readily omit specific arguments within sentences compared to English when these arguments have been previously referenced or can be inferred. As an example, when attempting machine translation from pro-drop languages such as Chinese and Korean, where pronouns are often dropped, into English, problems arise because English pronouns must be inferred from non-existent words to preserve meaning (Chung and Gildea, 2010). Therefore, recovering null elements from a parsed tree without traces may enable us to resolve null anaphora, where omitted pronouns or other referential expressions are understood from context. Outside of practical applications, null elements have a rich history in syntax, and are an essential aspect of explaining transformational grammars. By restoring null elements, parse trees may better reflect a linguistic understanding of movement, as well as many other linguistic phenomena relevant to the use of null elements.

Previous work has approached the recovery of null elements in different ways, ranging from more rule based approaches to statistical approaches. For example, Johnson (2002) introduced a pattern-matching algorithm aimed at recovering empty nodes within the Penn English Treebank (PTB) (Marcus et al., 1993). Their exploration involved scrutinizing parsing outcomes for various null elements such as *T* (WH trace) and *U* (empty units), demonstrating their potential to enhance parsing results. Campbell (2004) proposed a linguistically oriented approach grounded in Government-Binding theory (Chomsky, 1981), which forms the basis of syntactic annotation in treebanking. Despite the inclination towards a rule-based method for empty category retrieval, reported parsing outcomes have been less compelling (Campbell, 2004, p.650). Pattern matching algorithms have also been utilized to restore Restoring null elements for Chinese and Korean using a similar algorithm to Johnson, but included more context in the generated patterns to better recover null elements in Chinese, since Chinese contains null elements (*PRO* and *pro*) which have similar syntactic contexts so need wider patterns to be effectively recovered. (Chung and Gildea, 2010). In contrast, Levy and Manning (2004) introduced an algorithm based on statistical features for reconstructing non-local dependencies concerning null elements. Non-local dependencies are syntactic constructs where two words cannot be obviously related based on typical syntactic rules – null elements that are indexed to the dependant word are a way to represent non-local dependencies using existing syntactic rules.

Previous work on null element restoration has primarily focused on English – we hope to expand the scope of studies by implementing algorithms for null element recovery for English, Chinese, and Korean. Furthermore, we aim to take a a comparative approach by implementing linguistically-motivated algorithms for removing and restoring null elements, as well as expanding to a language agnostic neural experiments. Considering the general success of neural experiments in outperforming traditional statistical parsers, applying neural experiments to null element recovery may allow for greater accuracy than previous approaches.

After being introduced by Chomsky, null elements have become a standard feature of syntax and have been a topic of further exploration. Reeves (1992) discusses in depth the legitimacy and theory behind null elements in a purely theoretical context by generating a framework in which null elements are not allowed and testing if such a framework fails. Reeves concludes that while sometimes disruptive to work with, null elements are ultimately necessary for systematically representing and analyzing the English language.

Previous works on the computational approach to restoring null elements generally choose one of three methods for inserting null elements; pre-processing, in-processing, and post-processing. Such previous works and their respective approaches are noted in Table 1. We utilize the post-processing approach. In this section we give a broad overview of each approach and justify our decision to use post-processing.

As noted in Table 1, Dienes and Dubey (2003a) inserted null elements as a step before parsing (thus, preprocessing). However, preprocessing approaches to null element insertion have been linked to worse results for general parsing than with post-processing approaches (Johnson, 2002). Advantages of pre-processing versus post-processing or in-processing include runtime efficiency. (Dienes and Dubey, 2003a).

Other work utilized in-processing, making this the most popular approach from previous attempts of inserting null elements (Dienes and Dubey, 2003b; Schmid, 2006; Kato and Matsubara, 2016; Hayashi and Nagata, 2016; Kummerfeld and Klein, 2017). In in-processing, null elements are integrated into the parsing algorithm. Generally, in-processing results from previous work are relatively similar to post-processing results. However, in-processing is a more complex task than post-processing, and is less conducive to a rules based approach, since a rule-based approach to null element recovery uses existing syntactic structure to predict the position of null elements.

Finally, Johnson (2002) and Campbell (2004) utilized post-processing, during which null elements are inserted into the parse tree. Results from post-processing are favourable to some in-processing results Campbell (2004). Furthermore, post-processing avoids problems where the null element is inserted in the correct position, but there maybe bracketing errors higher up in the tree, such as in the grandparent of the null element. Preventing these types of errors simplifies the evaluation process, since we prevent the need for assessing the overall syntactic structure and can evaluate just the position and type of the null elements.

For this paper, we implement a post-processing approach (Johnson, 2002; Campbell, 2004). We believe that this approach might be the best for the insertion of null elements, as it has yielded promising results in previous studies without adding further complexities. However, this method has not been tested extensively in recent years, especially with neural approaches, including a language-agnostic sequence-to-sequence model.

The Berkeley Neural Parser has historically been used in processing trees from the Penn Treebank, and can be used to produce trees with and without traces. In trees without traces, null elements are thus removed. This approach works very well for English and Chinese to achieve the goal of removing null elements, but there is a knowledge deficit as there is no parser to remove null elements in Korean sentences. Furthermore, there have been no previous works attempting to restore null elements into Korean sentences. To address this knowledge gap, we have chosen to build off of the approach used by the Berkeley Neural Parser to similarly parse and remove null elements from Korean sentences, as well as use neural experiments to attempt the restoration of null elements into Korean sentences.

The Penn Treebank contains sentences sourced in English (Marcus et al., 1993), Chinese (Xue et al., 2005) and Korean (Han et al., 2002) – all of which contain null elements that work to communicate implicit syntactic structures. Table 2 in the appendix below shows which types of null elements appear in each language (English, Chinese and Korean). From this table we can see that English does not share many null elements with pro-drop languages such as Chinese and Korean; only traces of A’ movement (*T*) are present in all languages. Below, we also provide a brief description of each null element type – for further detail please refer to the Penn tree bracketing guidelines.

The removal of null elements has, in the past, been used as a step in prepossessing, since null elements have often primarily been seen as entities to remove. In particular, the Berkeley Neural Parser has been a useful tool in processing trees from the Penn Treebank. The Berkeley Neural Parser produces two outputs - trees with traces (with null elements) and without traces (null elements are removed). Examples of prepossessed trees, with trace and without trace, for English and Chinese are given in Figures 1 and 2, respectively.

The removal of null elements in both Chinese and English is very similar; the Berkeley Neural Parser simply removes the null element, and leaves its parent node as a single branching constituent. This removal sometimes violates the syntactic integrity of the clause, but creates a tree without such a null element. An example for English is given in Figure 1, in which the VP, "to see if advertising works" requires a null subject to represent the subject of the embedded clause to maintain its integrity. In removing this element, the Berkeley Neural Parser creates a sentence composed of just a VP, without a subject. In the Chinese sentence given in Figure 2, the embedded clause 从波黑撤军 cóng bō hēi chèjūn (‘withdraw troops from Bosnia’) has an arbitrary null subject, marked by the null element *PRO*. In order to remove this null element, the Berkeley Neural Parser creates an IP clause consisting of only a VP, without an NP subject. This is almost identical to the method used to remove the null subject in the earlier English sentence.

Besides the deletion of null elements, a difference between sentences with and without trace in both English and Chinese is that sentences with trace contain extra information about each node. For instance, "NP-SBJ" is written rather than just "NP", which is what is written on trees without traces. This difference causes trees with traces to contain more information about each constituent; namely, what role they perform within the clause they are a part of. When trees are generated without trace, this information is not kept track of. Hence, the second labels are removed.

Null element restoration has historically been approached in several ways – namely through a rule based, pattern matching, or statistical approach. Much of this study has been focused on recovering null elements in the PTB, but there has been studies also for the CTB and KTB, especially in the restoration of *PRO* and *pro*, as these elements have been seen as particularly relevant for machine translation tasks. Our study has been focused on replicating rule based approaches for the PTB, and generating novel rules to expand a rule based approach to the CTB. Furthermore, we seek to expand the set of available approaches to null element restoration by running neural experiments in the PTB, CTB, and KTB.

This study has improved on previous works by exploring more null elements in a rule-based approach for Chinese, as well as providing a basis for a language-agnostic neural approach for future works to expand on, which also explores null elements ignored by previous works. Ultimately, a neural approach may be the best suited to the task of null element recovery, considering that the neural experiments had comparable results to previous work for English and to our rule-based implementation for Chinese in elements other than *pro* and *PRO*. Although future work could improve upon our rule-based Chinese results to surpass our neural experiments, rule-based approaches are difficult to implement, and require a theoretical background in a given language. In contrast, language agnostic neural experiments can be more easily implemented across several languages and robustly handle null elements that previous work ignored. Furthermore, considering our use of a relatively simple architecture, future work in using seq2seq to recover null elements may produce results that compare or outperform rules-based approaches. More broadly, a neural approach to null element recovery is particularly useful for any languages that lack an existing language specific rule-based approach and for languages with a small number of null element types like Korean. With further work to improve on the baseline results for neural experiments set in this paper, neural results could be viable cross-linguistically as a general approach to null element recovery.

This study has primarily focused on the use of syntactic context to predict null elements. Future work interested in a linguistically oriented approach to null element recovery may benefit from the use of Semantic Role Labelling, or other tools that capture lexical information, to generate rules to predict some null element types. Furthermore, our neural experiments used linearized data which intentionally excluded lexical information, so that we could identify the efficacy of of neural networks in predicting null elements based on syntactic context. Future approaches may similarly enhance results by including lexical information.