Elementwise Language Representation

Most of the past and current state-of-the-arts and impactful studies in the field of natural language processing (Devlin et al., 2018; Radford et al., 2019; Lan et al., 2019; Yang et al., 2019; Brown et al., 2020; Clark et al., 2020; Raffel et al., 2020; Reed et al., 2022; Taylor et al., 2022) relies on subword-level tokenization (Sennrich et al., 2015; Wu et al., 2016; Kudo & Richardson, 2018). These pervasive choices are due to the reasonable trade-off between robustness and efficiency of subword tokenization, but their limitations in several special environments wherein the data is domain-specific and/or its distribution shifts frequently are problems that have to be addressed at some point. While some subsequent studies have suggested improved techniques for subword-level tokenization (Provilkov et al., 2019; He et al., 2020; Hiraoka et al., 2021; Wang et al., 2021), many of them involve significant increases in computational costs and engineering efforts.

Character-level modeling has long been proposed as a promising alternative to the (sub)word-level language representations. Although the chronic long-range dependence issues of pure character-level models (Sutskever et al., 2011; Graves, 2013; Zhang et al., 2015) solved by the adoption of the transformer architecture (Vaswani et al., 2017) as demonstrated in (Belouadi & Eger, 2022; Xue et al., 2022), the quadratic time-complexity of self-attention became a new bottleneck for character-level language representation. Some recently proposed studies try to address this problem by downsampling the long character sequences to an acceptable length: Tay et al. (2021) and Clark et al. (2022) utilize convolutional and non-parametric mean-pooling respectively, Godey et al. (2022) leverages non-parametric max-pooling. All of them requires additional computations for explicit downsampling and character-level enrichment, thus causing corresponding overheads. The proposed elementwise embedding achieves the similar using a trivial tensor reshaping operation so does not degrades the inference speed of its backbone transformer architecture while processing much longer sequences.

The major challenge of the transformer architecture Vaswani et al. (2017) is to mitigate its quadratic self-attention complexity. Liu et al. (2018) proposed to enhance this complexity by computing attentions within partitioned embedding matrices. Child et al. (2019) estimated the full attention by mixing a sparse number of local attentions. Beltagy et al. (2020) extended this idea using technique called dilated sliding window to cover a wider range of attention. Zaheer et al. (2020) achieved the similar by attending three different types of attentions: global, windowed and random. Kitaev et al. (2020) alleviated the memory complexity by applying locality sensitive hashing and reversible residual layers Gomez et al. (2017). Wang et al. (2020) improved overall attention complexity to be linear time by performing dimensionality reduction on the length axis, Katharopoulos et al. (2020) enhanced the computational complexity to linear time using kernel-based formulation and causal masking.

Approaches through other structural modifications have also been proposed. Lan et al. (2019) reduced the size of its BERT (Devlin et al., 2018) backbone extremely smaller by sharing weight parameters between every attention layer. The concept of knowledge distillation (Hinton et al., 2015) has been demonstrated to be useful by (Sanh et al., 2019; Tang et al., 2019; Jiao et al., 2019). Trials for improving the inference-time efficiency in which unimportant attention heads are pruned away (Voita et al., 2019; Michel et al., 2019); or "blocks" are pruned instead (Lagunas et al., 2021); were made. Some recent studies proposed to enhance computational complexity by downsampling input sequences to an acceptable length (Godey et al., 2022; Tay et al., 2021). Elementwise embedding is quite similar to these approaches in terms of increasing the efficiency of transformer architecture, but is fundamentally different in that it does not require any downsampling and architectural modification of the transformer model to work with. What it does is that simply projects $uv$ bytes sequence into a $(uv,c)$ character embedding matrix, reshapes it into a $(u,w=vc)$ embedding matrix, and pass it to a transformer as input features; thus making it to align $uv$ sequences at the $O(u^{2})$ complexity. As elementwise embedding is designed as a pure embedding technique that does not modify any part of the transformer architecture, it can be potentially utilized with all the above methods in conjunction.

Patent classification is an interesting subfield of text classification that targets to automize the categorization of patent documents. Although this research field has not been actively studied compared to other compelling applied-ml areas like medicine and robotics, it deserves attention because it is essential for the data-driven patent analysis (Lee et al., 2009; Kim & Lee, 2017). Patents filed during a specific period are often closely associated with technological trends at that time which implies a big flow of capital, and modern machine learning methods as large language models (Brown et al., 2020; Taylor et al., 2022) and graph-powered neural networks (Sanchez-Gonzalez et al., 2020; Jumper et al., 2021; Stokes et al., 2020) are powerful enough for extracting meaningful patterns from those textual/tabular data to lead high-impact decision makings.

In addition to the purpose of patent analysis, patents are also valuable for general-purpose machine learning research. Patent documents are large amounts of multi-modal data that consist of texts, graphs, images and their organized structure makes it easier to preprocess than dealing with raw data such as randomly crawled web corpora. Furthermore, patent data can be used to benchmark learning algorithms because its extremely imbalanced distribution and a wide range of domain-specific lexicons hinder the models from convergence. This is the main reason that we utilize patent classification for evaluating improved robustness of our models trained with the proposed elementwise embedding.

The technical requirement of patent classification is largely threefold. First, the classifier should be possible to capture the unique properties (i.e., the classification symbols) of each patent that is computed by Precision and must be able to distinguish between different patent documents (i.e., the difference between classification symbols of two separate patent documents) that is calculated by Recall. Second, the classifier has to do well on both Precision and Recall, and every classification symbol should have equal importance (all technical categories are potentially important even if it is not currently popular). The former can be achieved by F₁ measure which is the harmonic mean of Precision and Recall, and the latter is satisfied by computing the micro-averaged scores for these three metrics. Third, the first-listed classification symbol must best indicate the invention of each patent and later ones offer additional information regardless of their relative positions (i.e., the meaning of a symbol differs by its order listed). To the best of our knowledge, however, this guide has not been reflected in known previous studies on multilabel patent classification (Lim & Kwon, 2017; Li et al., 2018; Yadrintsev et al., 2018; Lee & Hsiang, 2020; Haghighian Roudsari et al., 2022). They considered two different patents having classification symbols [G06Q, G06Q, A01B], [A01B, G06Q, A01B] as identical, one-hot encoding their labels as [A01B: 1, G06Q: 1]. This skewed labeling no longer provide classifiers with the correct evaluation criteria. To fix this problem, we performed simple relabeling that conserves the order of class labels (see Section 5.3) and compared the experimental results with our own baselines. Section 5.5 offers the equations of the above three metrics used in the following experiments.

where $\mathrm{H}$ is the Shannon entropy $\mathrm{H(x)}=-\mathbb{E}_{\mathrm{x}\sim p}[\mathrm{log}P(x)]$ and $v\in\mathbb{R}$. Because low entropy means there are fewer cases to encode, it is natural to represent a character as a much lower-dimensional latent embedding. Assuming a character as one of UTF-8 bytes ¹¹1While only ASCII characters can be expressed in 1 byte, we used this analogy for ease of explanation. and each semantic unit (i.e., a token) consists of $v$ characters, a neural network with $w$-dimensional hidden layers will have 256 $c$-dimensional character embeddings as its embedding table when $c=w/v$. A semantic unit is abstracted into a horizontal concatenation of $v$ character embeddings. Larger meanings are just concatenations of smaller ones and this hierarchical expression allows neural networks to explicitly recognize characters while learning at any complex-level of semantics. We name this pair of $(256,c)$ embedding table and the following concatenation operation as elementwise embedding, referring 256 character embeddings to elements and their concatenated meanings to materials.

Note that by matching the number of attention heads with $v$, we can restrict the subspace of each attention head to the latent space of $c$-dimensional character embeddings (i.e., elements), when attention layers expect $vc=w$-dimensional embeddings (i.e., materials) as input features. It can be interpreted as approximating the $w$-dimensional latent space with closed set of $w/v=c$-dimensional vectors, and also as separating the roles of embeddings and hidden layers: embeddings to encode character-level semantics and hidden layers to encode more complex-levels of semantics. This helps neural networks do not waste their limited expressiveness on encoding implicit information. Because the transformer architecture (Vaswani et al., 2017) is itself multiple layers of attention, we do not have to implement $focus$ operation by hand, so what we need for elementwise language representation is only a single-time tensor reshaping operation from $(uv,c)$ to $(u,w)$; which is the same as concatenating $uv$ c-dimensional elements to be $u$ $w$-dimensional materials. Following this framework, the standard transformer architecture can align $uv$ characters at the $O(u^{2})$ computational complexity fully ignoring the value of $v$. The acceptable length of the input sequence scales with the size of the hidden layers so larger models can process longer sequences than the smaller ones at the same attention complexity; this is more natural than the current transformers that no matter how large the model is, the length of the sequence it can process in a reasonable amount of time does not change at all.

In practical implementation, focus for aligning important elements is performed by the mutli-head attention of parent architecture (i.e., a transformer model to work with) having $h=v$ attention heads, so elementwise embedding is just a set of lookup table containing 256 $c$-dimensional element embeddings and the following tensor reshaping operation. In other words, applying elementwise embedding is simply to replace the existing embedding table of parent model with elementwise embedding. While token embeddings are usually trained with neural networks, they can be always detached fully independently of the network architecture so that we do not regard replacing embedding table as a structural modification. This feature is clearly different from the previous studies on character-aware language representation that additional computational components (e.g., non-parametric mean/min/max pooling, shallow convolutional/recurrent/transformer layers) are required for enriching character-level information or up/downsampling input sequences as leveraged in (Ma et al., 2020; Tay et al., 2021; Clark et al., 2022; Godey et al., 2022).

Technically, elementwise embedding is implemented by a single-time reshaping operation as,

We add two kinds of position embeddings $g^{(p)}$ and $f^{(i)}$ called "focus embeddings" to $e^{(i)[j]}$ and $e^{(i)}$ respectively, to manually encode the focusable positions: the former describes global positions (e.g., position of a character $e^{(i)[j]}$ in the entire sentence $e_{u}$) and the latter directs the local positions (e.g., position of a spelling $e^{(i)[j]}$ in the $i^{th}$ word $e^{(i)}$). Though elementwise embedding works well without focus embeddings, we found that they help models trained with elementwise embedding a lot to converge more stable and to perform better as explained in Section 6.1. Before being passed to as input to the parent model, dropout (Srivastava et al., 2014) and normalization (Ba et al., 2016) can be applied to embeddings for better generalization performance.

Elementwise embedding applies with any type of tokenization by following the framework:

1. 1.

   Divide given text into $u$ tokens

2. 2.

   Encode each token into $v$ integers (UTF-8 bytes)

3. 3.

   Project integers into a $(uv,c)$ element embedding matrix

4. 4.

   Reshape element embedding matrix into a $(u,w)$ material embedding matrix

where $w=vc$ (see Fig 2).

All models used in the following experiments were trained on patent documents published by the USPTO (United States Patent and Trademark Office) from 2006 to 2014 and then tested on the two test set splits 2015_A and 2015_B that consist of patents in the first- and second-half of 2015, respectively. We leveraged each patent document as a single training example: its claim texts as input features and the classification symbols (i.e, CPC codes) assigned to it as labels. Among the five hierarchical levels of Section, Class, Subclass, Main Group, Subgroup, we used the slices from Section to Subclass (i.e., subclass-level CPC codes) as labels: label of the CPC code A01N 53/12 [Section: A, Class: 01, Subclass: N, Main Group: 53/00, Subgroup: 53/12] is A01N which is the concatenation of slice [Section: A, Class: 01, Subclass: N]. We concatenated the first 20 claims into a single input text, instead of using the first claim only as in Lee & Hsiang (2020) to prevent the classifier from overfitting on meaningless training examples that are too short to describe each patent’s own invention. The entire number of class labels are doubled from 664 from 1,328 by the two position attributes First and Later after relabeling, decreasing the standard deviation of every dataset split (see Section 5.3) which means that the imbalances between class labels are quite mitigated. Because we used patent data to benchmark the robustness of classification models to domain-specificity and long-tailed distribution, we did not adjust the dataset imbalance further. We collected utility type patents only filed in the United States as training data from the BigQuery table called Google Patents Public Data ²²2https://github.com/google/patents-public-data. Table 2 provides a statistical summary of the dataset.

As mentioned repeatedly in previous sections, applying elementwise embedding is to replace the embedding table of the parent transformer architecture. We trained three different transformers BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), CANINE (Clark et al., 2022) with elementwise embedding for multilabel patent classification and compared with their original implementations to demonstrate the idea of elementwise language representation. We denote three transformers with elementwise embedding by EWE models such as BERT_EWE, ALBERT_EWE, CANINE_EWE and their original versions, the baselines, as ORIG models like BERT_ORIG, ALBERT_ORIG, CANINE_ORIG. In both theoretical and practical implementations, elementwise embedding does not modify any part of its parent model so that EWE models always have exactly the same architectures as their ORIG equivalences. Every model used in our experiments follow the configuration of BERT_BASE (Devlin et al., 2018) wherein a transformer encoder has 12 768-dimensional attention layers with 12 heads ³³3While EWE models use 16 attention heads, this is only to meet the theory of elementwise embedding so that is independent of their superior performances to the ORIG models as shown in the ablation study in Section 6.2 each followed by a 3072-dimensional linear projection; every model processes $u=128$ tokens at once but EWE models align $v$ times longer sequence than ORIG models since each token is represented by $v$ embeddings (see Fig 3). Table 1 shows the differences in between configurations of EWE and ORIG models.

For the purpose of patent classification, the label of each patent document becomes a list of textual symbols classifying it to the corresponding technical categories. Among several classification schemes, the two most frequently utilized from previous studies are IPC (International Patent Classification) and CPC (Cooperative Patent Classification). One interesting fact is that the assigned symbols have different meanings depending on the order in which they are listed: the first-listed symbol describe the invention of each patent document and the later symbols represent additional information, so that patent documents with classifications [G06Q, G06Q, A01B], [A01B, G06Q, A01B], [G06Q, A01B] should have three different labels. This is in accordance of their documentations (USPTO, 2022; WIPO, 2022), but to the best of our knowledge, previous studies on patent classification (Lim & Kwon, 2017; Li et al., 2018; Yadrintsev et al., 2018; Lee & Hsiang, 2020; Haghighian Roudsari et al., 2022) did not reflect this guide, hence one-hot encoding the above three patents to an identical label [A01B: 1, G06Q: 1] (see the left of Fig 4). This skewed labeling is quite undesirable for both practical patent classification and algorithm benchmarking since the metrics for evaluating the classifier will be messed up by the distorted one-hot encodings.

To address this problem, we simply relabeled our patent documents by attaching position attributes First and Later as prefixes to their classification symbols. We relabeled the above classifications [G06Q, G06Q, A01B], [A01B, G06Q, A01B], [G06Q, A01B] as [First-G06Q, Later-G06Q, Later-A01B], [First-A01B, Later-G06Q, Later-A01B], [First-G06Q, Later-A01B] so that to be one-hot encoded as [0, 1, 1, 1], [1, 0, 1, 1], [0, 1, 1, 0]; when the placeholder for one-hot encoding is [First-A01B, First-G06Q, Later-A01B, Later-G06Q] (see the right of Fig 4). Relabeled symbols fully satisfy the technical requirements for patent classification described in Section 3.3 and furthermore, the imbalanced distribution between class labels were significantly alleviated since the count of each classification symbols (CPC codes here) was halved by position attributes First and Later (See Table 2). As a result, all our models trained on relabeled data do well both on Precision and Recall (see Table 3), compared to the case of aforementioned studies that are biased against either score.

This section provides specific configurations to train our models. Every model was trained during 10 epochs using binary cross-entropy loss with sigmoid activation ($threshold=0.3$) for multilabel patent classification from scratch without pretraining. We used AdamW (Loshchilov & Hutter, 2017) as optimizer ($\beta_{1}=0.9$ and $\beta_{2}=0.999$, $eps=1e-8$) with $L_{2}$ regularization ($\lambda=0.01$). Initial learning rate decays linearly from $2e-5$ without a warmup period and batch size was set to 32; we selected these small values to prevent overfitting due to imbalanced training examples. Larger memory-safe batch sizes did not show meaningful differences, only slowing convergence. All models were trained and tested using a single 24GB VRAM GPU (NVIDIA RTX TITAN) with FP16 mixed-precision.

For evaluating models for multilabel patent classification, we utilize three metrics Precision, Recall and their harmonic mean F₁ measure. Because all CPC codes (technical categories of patents; class labels) are equally important (current technical categories draw long tail when counted by the number of the patent documents that they are classifying; only 30% of the categories contain over 90% of the entire patents, mainly due to the popularity of each field of technology; but all of them are potentially valuable) we compute micro-averaged scores (TP, TN, FP, FN) for all metrics by $\mathrm{Score}=\sum\nolimits_{l}\mathrm{Score}_{l}$. Since we consider a total of 664 CPC symbols as class labels with two position attributes First and Later, $l\in[1,1328]$ is established. As mentioned in Section 3.3, $\mathrm{Precision=TP\cdot(TP+FP)^{-1}}$ captures how well a model identifies the unique classifications of each patent document, $\mathrm{Recall=TP\cdot(TP+FN)^{-1}}$ implies how well a model distinguishes between different patent documents which is critical for patent search engine optimization. As Precision and Recall are equally important for patent classification, we utilize their $\mathrm{F_{1}=2\cdot(Precision^{-1}+Recall^{-1})^{-1}}$ as the main metric.

Originally, the main design goal of focus embedding was to stabilze the convergence of models trained with elementwise embedding. While the idea of elementwise embedding works well without focus embeddings (see Table 4), training parent models with it was somewhat tricky; the starting point of meaningful convergence differed randomly at each training trial, and therefore the reproducibility was not ensured. Focus embeddings guarantee the stable convergence of elementwise models by explicitly encoding the global and local positions of focusable characters. We found that focus embeddings also improve classification performance of parent transformer, so set them as the default component of elementwise embedding. In the case where the positional information is supplemented in some other way e.g, $n$-gram as in tokenization-free BERT_EWE (see Section 6.2) focus embedding did not provide a meaningful enhancement in both stability and performance, however, more research is needed on how their absence will affect other applications as sequence-to-sequence modeling. Table 4 shows the ablation study on focus embeddings BERT_EWE classifier.

In this section, we generalize the idea of elementwise embedding to tokenization-free language representation. We implement two versions of tokenization-free BERT_EWE: one using pure UTF-8 byte-level tokenizer⁵⁵5UTF-8 bytes encoding can be achieved by list(bytes("text/to/encode", "utf-8")) in Python3. and one using gradient-based tokenizer. Tokenization-free elementwise language representation follows the same framework described in Fig 2 and Section 4.2 ⁶⁶6For tokenization-free elementwise language representation, text is encoded to a sequence of $uv$ UTF-8 bytes, projected into a $(uv,c)$ element embedding matrix, then reshaped into a $(u,w)$ material embedding matrix directly without any tokenization.. First, BERT_EWE trained using raw UTF-8 bytes encoding (see the first row of Table 5) shows significantly poor performance compared to one trained using whitespace tokenizer (see the third row Table 5), but still equivalent to CANINE_ORIG which is 1.3x larger. This version of BERT_EWE shares the same hyperparameters with whitespace version which set $n=v=16$ and $u=128$.

Notably, tokenization-free BERT_EWE recovers the same level of performance as one with whitespace tokenizer when trained using gradient-based tokenizer (see the second row of Table 5). In this implementation, each element is replaced with softmax-weighted sum-pooled $v$-gram⁷⁷7This can be understood as the simplified version of Block scoring network proposed in Tay et al. (2021).:

where $\alpha_{j}=\mathrm{softmax}_{j}(e^{(i)[j]}s)$ and $s\in\mathbb{R}^{c}$ is the weight vector for linear projection $\mathbb{R}^{c}\mapsto\mathbb{R}$. Since aggregated $v$-grams implicitly encode the positions of focusable elements, focus embeddings are not used for this model. $n=v=8$ attention heads are used. Operations for $v$-gram pooling cause overhead but is trivial, so that the resulting decrease in inference speed is negligible. This type of elementwise representation is expected to be useful for the tasks where consistent tokenization is difficult (e.g., multilingual applications which deal with heterogeneous language systems simultaneously). For large language models such as GPT-3 (Brown et al., 2020) with much wider hidden layers (so each semantic unit can have much greater numbers of characters), however, whitespace tokenization will suffice for almost all kinds of language representations.