Sequence Model with Self-Adaptive Sliding Window for Efficient Spoken Document Segmentation

Figure 1(a) illustrates the proposed SeqModel, which models document segmentation as a sentence-level sequence labeling task. Each sentence in a document is first tokenized by WordPiece tokenizer [12]. A block of consecutive sentences $\{s_{m}\}_{m=1}^{n}$ is prepended with a special token [CLS]. This sequence is embedded through the input representation layer as the element-wise sum of token embedding, position embedding, and segment embedding. These embeddings are then fed to a transformer encoder and the output hidden states corresponding to $k$ tokens in sentence $s_{m}$, denoted by $\{R_{m-i}\}_{i=1}^{k}$, are mean-pooled to represent the sentence encoding. The sentence encodings $\{S_{m}\}_{m=1}^{n}$ for sentences $\{s\}_{m=1}^{n}$ are fed to a softmax binary classifier to classify whether each sentence is a segment boundary. The training objective is minimizing the softmax cross-entropy loss.

One SOTA model, the cross-segment BERT model [10], treats document segmentation as a token-level binary classification task for each candidate break (i.e., each token). The input training sample is composed of a leading [CLS] token, and concatenation of the left and right local contexts for a candidate break, separated by the [SEP] token. This sequence is fed to a BERT-encoder and the output hidden state corresponding to [CLS] is fed to a softmax classifier for segmentation decision on the candidate break. In comparison, our SeqModel encodes a block of sentences $\{s_{m}\}_{m=1}^{n}$ simultaneously, hence exploits longer context and inter-sentence dependencies through encoder self-attention. Consequently, the contextualized sentence encodings $\{S_{m}\}_{m=1}^{n}$ for classification can potentially improve segmentation accuracy. In addition, our SeqModel classifies multiple sentences simultaneously, hence significantly speeds up inference compared to the cross-segment model. Also, different from another SOTA model Hier.BERT [10], SeqModel generates contextualized sentence encoding through mean-pooling, instead of employing another transformer encoder for document encoding as in Hier.BERT, hence significantly reduces computational complexity and improves inference efficiency.

Document segmentation requires deep understanding of document structure. We hypothesize that using pre-trained models with enhanced structural modeling for SeqModel may improve segmentation accuracy. BERT [13] employs the next sentence prediction (NSP) task as an inter-sentence objective, which is a binary classification task deciding whether two sentences are contiguous in the original source. RoBERTa [14] removes the NSP objective from pre-training but also adds other training optimizations. StructBERT [15] augments BERT pre-training objectives with a word structural objective and a sentence structural objective. The sentence structural objective conducts a ternary classification on two sentences $(S_{1},S_{2})$ to decide whether $S_{1}$ precedes or follows $S_{2}$ or the two sentences are noncontiguous. The CONPONO model [16] exploits new inter-sentence objectives, including predicting ordering between coherent yet noncontiguous spans with multiple sentences in between. ELECTRA [17] replaces masked language modeling in BERT pre-training with a more efficient replaced-token detection task, which replaces some input tokens with plausible alternatives generated by a generator network, then trains a discriminator to predict whether each token in the corrupted input is replaced by a generated sample or not. We include RoBERTa and ELECTRA in investigation due to their strong performance, but they do not enhance structural modeling particularly.

We also propose a self-adaptive sliding window approach to further speed up inference without sacrificing accuracy, as depicted in Figure 1(b). Traditional sliding window for segmentation inference uses a fixed forward step size. In our proposed self-adaptive sliding window approach, during inference, starting from the last sentence in the previous window, the model looks backward within a maximum backward step size to find positive segmentation decisions (segmentation prediction probability from the model $>0.5$) from the previous inference step. When there are positive decisions within this span, the next sliding window is automatically adjusted to start from the next sentence after the most recent predicted segment boundary. Otherwise, when no positive decisions exist within this span, the next sliding window starts from the last sentence in the previous window. Considering that the last paragraph and the history beyond have reduced impact on the next segmentation decision, this strategy helps discard irrelevant history within the sliding window. Hence, the self-adaptive sliding window may both speed up inference and improve segmentation accuracy.

For training, we compare two approaches: constructing samples following the self-adaptive sliding window approach, but using the segmentation references instead of predicted positive segmentation decisions from the model; or constructing samples with fixed sliding window. Compared to using the self-adaptive sliding window with segmentation references, the fixed sliding window approach is more effective at reducing the discrepancy between training and inference, creates more training samples, and ultimately ameliorates robustness of the segmentation model during inference to noises from wrong segment boundary predictions from the model. We observe training with fixed sliding window significantly outperforms training with self-adaptive sliding window using segmentation references, on segmentation F₁. In order to make SeqModel learn inter-sentence dependencies among as many sentences as possible and become robust to sentences with missing tokens, for training sample sequences longer than the max sequence length, we first truncate them beyond the max sequence length. We also create an alternative by truncating tokens in sentences of the sequence while keeping the max sentence number in the sequence. We pool these two versions for training. For training SeqModel, we optimize forward step size for fixed sliding window and the max sentence number, based on segmentation F₁ on Wiki-727K dev.

Inputs to SeqModel for spoken document segmentation are ASR 1-bests. ASR errors usually comprise acoustically confusing words with similar pronunciations yet different meanings, which become noise to the input embedding layer of SeqModel. We propose an approach to improve robustness to ASR errors by augmenting the input embeddings with phone embeddings (denoted by PEs), as shown in the input embedding layer in Figure 1(a). PEs also serve a regularization to the original input embedding. Details for computing PEs are illustrated in the enlarged view of the phone component in Figure 1(a)²²2We compared computing the PEs as the sum or the mean of sub-phone embeddings and observed consistently better segmentation accuracy from mean-pooling.. Given a word (use the first word $w_{1}$ in $s_{1}$ as an example) and its phone sequence $\{p_{1-1-k}\}_{k=1}^{j}$ (from looking up an in-house Chinese pronunciation dictionary and picking up the first entry for the word), the sub-phone embeddings are denoted by $\{P_{1-1-k}\}_{k=1}^{j}$. The augmented input embedding $E[w_{1},\{p_{1-1-k}\}_{k=1}^{j}]$ for $w_{1}$ is computed as $E[w_{1},\{p_{1-1-k}\}_{k=1}^{j}]=E[w_{1}]+\frac{1}{j}\sum_{k=1}^{j}E[p_{1-1-k}]$, where $E(*)$ denotes trainable embedding; $E[w_{1}]$ denotes the standard BERT-style input embedding. In Figure 1(a), $\{P_{1-1-k}\}_{k=1}^{j}$ denotes sub-phone embeddings $\{E[p_{1-1-k}]\}_{k=1}^{j}$; $A_{1-1}$ denotes $\frac{1}{j}\sum_{k=1}^{j}E[p_{1-1-k}]$.

We conduct document segmentation evaluations on three datasets, including the commonly used English Wiki-727K benchmark, a Chinese Wikipedia-based document segmentation dataset that we created in this work (denoted by Wiki-zh), and an in-house Chinese spoken document dataset.
Wiki-727K and Wiki-zh. The Wiki-727K dataset [7] comprises 727K English Wikipedia articles. We use the same train/dev/test partitions as in the original work [7]. Note that English Wiki-727K uses the hierarchical segmentation of articles as in their table of contents, that is, the segment boundaries are section boundaries. Due to lack of Chinese document segmentation benchmarks, we create the Wiki-zh dataset in a similar procedure as used for creating Wiki-727K except one difference. Given that Chinese wikipedia articles are much shorter than English wikipedia articles³³3https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/, for Wiki-zh, to make the average number of segments per document comparable to that of Wiki-727K, we define segment boundaries as paragraph boundaries. From a Chinese Wikipedia snapshot, we extract and tokenize documents and define reference segment boundaries based on double new lines.
Building the Chinese spoken document dataset. Public spoken document segmentation datasets are particularly scarce. Our in-house Chinese spoken document dataset consists of two subsets as single-speaker lectures (denoted SD-zh-SP) and multi-speaker interviews (denoted SD-zh-MP), all from the finance domain. We use an in-house open-domain ASR system (CER 16.0% on in-house testsets) to generate ASR 1-best for SD-zh-SP and SD-zh-MP. We employ transformer-based punctuation prediction to insert period, question mark, and comma into the ASR 1-bests. The resulting transcripts are used for manual annotations through crowd-sourcing, by annotating whether each sentence (based on the automatically inserted period or question mark) is at the paragraph boundary. However, these ASR 1-bests contain ASR errors, ungrammatical structures, disfluencies , etc. All these issues pose severe challenges to manual annotations. We observe significant inter-annotator disagreements on paragraph annotations. Initially, only 5% annotated sentences have the same labels from multiple annotators.

We tackle this challenge with a three-stage approach, namely, screening, annotation, and post-annotation filtering. During screening, the annotators are asked to annotate paragraphs for formal news text (with reference paragraph boundaries removed). Annotations from the annotators are scored against the reference paragraph boundaries. Only annotators with F${}_{1}>60$ are selected for annotating the spoken documents. During annotation, we collect annotations from 5 annotators for each sentence. During post-annotation filtering, we use leave-one-out to score the annotations from each annotator. The top4 annotations for each sentence are used for majority voting to make the final segmentation decision for each sentence ($\geq 3$ positive votes count a positive decision). We observe significant improvement on annotation quality from this approach. On formal news text, the average annotation F₁ improves from 61.8 to 68. On the combined set of SD-zh-SP and SD-zh-MP, the average F₁ from majority voting on 5 annotations is 33.4. Using the top4 annotations after leave-one-out improves F₁ to 39.5. Still, the gap between the average F₁ on written text and spoken documents indicates the challenges for spoken document segmentation annotations, which demands more studies.

Comparison with previous SOTA. Table 2 shows P, R, and F₁ on the Wiki-727K test set comparing our SeqModel with prior models. The results for prior models, including BERT+Bi-LSTM, Hier.BERT and Cross-segment BERT-Base/BERT-Large with different left-right context lengths (128-128, 256-256), are all cited from [10] and the best previous F₁ 67.1 on Wiki-727K is from Cross-segment BERT-Large 256-256. We reimplemented Cross-segment BERT-Base and replicated results on the Wiki-727 test set as in [10]. As can be seen from the table, the proposed SeqModel based on BERT-Base (denoted SeqModel:BERT-Base, 110M parameters) establishes a new SOTA F₁ 68.2, 4.2 points gain over Cross-segment BERT-Base 128-128 (F₁ 64.0) with the same number of parameters, and 1.1 points gain over Cross-segment BERT-Large 256-256 with much larger 336M model parameters. Table 3 compares the segmentation accuracy from SeqModel:BERT-Base and Cross-segment BERT-Base 128-128 on the Wiki-zh test set, SD-zh-SP, and SD-zh-MP. Again SeqModel:BERT-Base significantly outperforms Cross-segment BERT-Base 128-128 on the three Chinese test sets, by 4.3 (69.4 to 73.7), 8.8 (21.0 to 29.8), and 10.1 (13.2 to 23.3) points gains on F₁, respectively. All these improvements are statistically significant with $p<0.0005$.

Table 4 shows P, R, and F₁ from SeqModel using different pre-trained model MODEL, denoted by “SeqModel:MODEL”. Confirming our hypothesis, SeqModel:StructBERT-Base, based on StructBERT with enhanced modeling of inter-sentence relations, outperforms SeqModel:BERT-Base by +2.2 F₁ on Wiki-727K, +2.1 F₁ on Wiki-zh, +7.5 F₁ and +1.3 F₁ on SD-zh-SP and SD-zh-MP, respectively. There is only a minor +0.2 F₁ gain from SeqModel:CONPONO-Base over SeqModel:BERT-Base on Wiki-727K, although CONPONO-Base also extends inter-sentence objectives. Note that since a Chinese CONPONO pre-trained model is not available, no results are reported for SeqModel:CONPONO-Base on the three Chinese datasets. Although RoBERTa removes NSP from pre-training, SeqModel:RoBERTa-Base yields higher F₁ than SeqModel:BERT-Base on all testsets, due to other training optimizations. It is noticeable that although ELECTRA does not particularly enhance structural modeling, SeqModel:ELECTRA-Base attains the best F₁ on Wiki-727K and SD-zh-MP and second best F₁s on Wiki-zh and SD-zh-SP testsets, indicating that the more challenging and efficient pre-training task in ELECTRA is beneficial for SeqModel.

Effect of adding phone embeddings. The effect of adding phone embeddings (denoted by +phone) on spoken document segmentation accuracy is shown in Table 5. On the two spoken document datasets SD-zh-SP and SD-zh-MP, SeqModel:BERT-Base+phone notably outperforms SeqModel:BERT-Base by 2.8 and 2.1 points gains on F₁, both gains being statistically significant. For SeqMode:StructBERT-Base, +phone does not obtain improvement on SD-zh-SP but obtains +0.8 F₁ gain on SD-zh-MP. We observe that after fine-tuning with +phone, the input embeddings of words with similar pronunciations become close. Hence, this approach significantly improves robustness of the input embedding layer to ASR errors, and in turn, robustness of SeqModel.

Effect of self-adaptive sliding window. We randomly sample 1K documents from the Wiki-727 test set and measure the average inference time (ms) per document⁵⁵5Machine for inference is Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz. Intra_op_parallelism_threads and inter_op_parallelism_threads are set to 8 and 2, respectively.. Figure 2(a) and  2(b) show F₁ and inference time for SeqModel using different max backward step sizes (shortened to step sizes) for fixed and self-adaptive sliding window during inference. For fixed sliding window, the next window starts from the last sentence in the previous window - max backward step size + 1, to be comparable to self-adaptive sliding window. For both sliding windows for SeqModel, the window size is 512 tokens. Cross-segment BERT-Base (with contexts 128-128) can be considered as using step size 1 and window size 256, and its inference time is 4169 ms/doc. SeqModel with fixed sliding window-step size 5 achieves 762 ms/doc, 81.7% relative inference speedup. As analyzed in Section 2.2, self-adaptive sliding window dynamically discards irrelevant history for segmentation. When the step size increases, fixed sliding window includes more irrelevant content, hence we observe significant increase in inference time and drastic F₁ degradation (step size beyond 5). In contrast, for self-adaptive sliding window, when the step size increases, the increase in inference time is much smaller and slower, and F₁ improves slightly yet steadily. These results demonstrate the advantages of self-adaptive sliding window on inference efficiency and accuracy. Replacing fixed sliding window-step size 5 by self-adaptive sliding window-step size 10 further reduces SeqModel inference time to 689 ms/doc, less than 1/6 of inference time of cross-segment BERT-Base and a further 9.6% relative speedup, while retaining the same F₁.

Readability Evaluations. Spoken document segmentation aims at improving readability of ASR transcripts. Inspired by [19, 20], we design a reading comprehension evaluation since reading comprehension efficiency and accuracy strongly correlate with readability. We randomly sample 8 documents from SD-zh-SP and design 5-8 (on average 6) single-choice questions for each document based on its content. Every document $D_{i}$ is segmented by each segmentation model $M_{j}$. The segmented document $D_{i}^{j}$ by applying $M_{j}$ on $D_{i}$ creates 50 crowd-sourcing jobs, in total $8\times 50=400$ jobs for $M_{j}$, and each worker only works on documents segmented by one model. For each $M_{j}$, we select effective returns as returns with all questions for one document answered and the recorded duration $d$ for answering all questions satisfying $6$ minutes $\leq d\leq 20$ minutes. We obtain on average 87 effective returns among the 400 jobs for each $M_{j}$, probably due to the challenges posed by long-form spoken documents and the complex finance domain. The average number of answers to compute answer accuracy for each model is $87\times 6=522$ samples. SeqModel:BERT-Base demonstrates better readability by achieving 64.5% answer accuracy, compared to 59.5% from cross-segment BERT-Base.