Towards trustworthy phoneme boundary detection with autoregressive model and improved evaluation metric

SuperSeg consists of three main parts; acoustic encoder, boundary embedder, and boundary decoder. Acoustic encoder extracts latent features $\textbf{h}_{1},...,\textbf{h}_{T}\in\mathbb{R}^{d_{h}}$ from the mel-spectrogram of a given speech where $T$ denotes the total frame length. More specifically, acoustic encoder first transforms a log-scale mel-spectrogram of dimension $d_{mel}$ to $d_{l}$-dimensional variables $\textbf{m}_{1},...,\textbf{m}_{T}\in\mathbb{R}^{d_{l}}$ ($d_{l}\geq d_{h}$) and processes them using convolutional neural networks (CNNs) consisting of ReLU activations, layer normalization [18], and dropout layers [19]. Boundary embedder maps binary values of boundary (1 if the current frame contains a phoneme boundary, 0 otherwise) to $d_{e}$-dimensional vectors $\textbf{e}_{1},...,\textbf{e}_{T}\in\mathbb{R}^{d_{e}}$. Boundary decoder employs a unidirectional LSTM [20] that receives as inputs the $t$-th latent feature $\textbf{h}_{t}$ and the previous boundary embedding vector $\textbf{e}_{t-1}$, and outputs a Bernoulli parameter $p_{t}$ that estimates the probability that the current frame contains a phoneme boundary. Fig. 1 shows the detailed architecture of SuperSeg.

SuperSeg is optimized to solve binary classification tasks using the binary cross entropy (BCE) loss. To satisfy the autoregressive nature, the boundary embedding vectors obtained from the ground-truth labels are shifted to the right by 1 time step, and passed to boundary decoder (i.e., teacher forcing). For better generalization, we adopt data augmentation algorithms such as pitch/formant perturbations [12] and masking blocks of frequency channels [13].

At test time, SuperSeg sequentially identifies phoneme-level boundaries on a frame-by-frame basis. To decide the label of the $t$-th frame, SuperSeg compares the model output $p_{t}$ with the threshold $\nu$ (1 if $p_{t}>\nu$ and 0 otherwise). The threshold $\nu$ is determined based on the evaluation metric (e.g., R-value [17]) with the grid search using the validation set before evaluating on the test set.

In the conventional calculation of precision (P) and recall (R) [7, 9, 15], boundaries are usually evaluated not as a sequence but as individual elements, which overrates the repeated estimates around the true boundaries. We note that this often leads to unreliable scores of F1 and R-value since they are computed by using precision and recall. To tackle this issue, we propose to measure precision and recall by evaluating each boundary sequentially as shown in Alg. 1. The proposed algorithm avoids multiple but redundant contributions, thus giving a low score for over-segmentation. We use the true and detected boundaries for $A$ and $B$ respectively to get the recall, and swap the order to calculate the precision. The F1-score and R-value are obtained from these precision and recall values in the same way as before. Fig. 2 demonstrates the conventional and proposed hit counting methods.

We used as input a log-scale 80-channel mel-spectrogram ($d_{mel}=80$) that is computed on 40-millisecond windows with a stride of 10 milliseconds. The output dimension of the initial linear layer was set to 256 ($d_{l}=256$), and the dimension of the latent feature $\textbf{h}_{t}$ was set to 192 ($d_{h}=192$). We used 6 blocks ($N_{l}=6$) consisting of a convolution network, layer normalization, ReLU activation, and a dropout layer with the zeroing probability of 0.4. The kernel size of convolution networks was set to 3 ($k=3$) and we used the dilation cycle of [1, 2, 4, 1, 2, 4]. The dimension of boundary embedding vector $\textbf{e}_{t}$ was set to 64 ($d_{e}=64$). Lastly, the unidirectional LSTM used hidden states with the size of 256 and transformed them into one-dimensional Bernoulli parameters.

We conducted experiments using two popular benchmarks for phoneme boundary detection; TIMIT [10] and Buckeye [21] corpora. For the TIMIT dataset, we split the original training set into training and validation sets at a ratio of 9:1. The test split of the TIMIT dataset was used as is. For the Buckeye dataset, we constructed training, validation, and test sets by splitting the whole data at a ratio of 8:1:1 based on speaker identities following Kreuk et al. [15].

We trained SuperSeg on the TIMIT and Buckeye datasets using the AdamW optimizer [22] with a learning rate of $0.0005$ up to 1600 and 1000 epochs, respectively. The training was performed on a single RTX 3090 GPU with a batch size of 256. We employed data augmentation methods to reduce overfitting. For frequency masking, we uniformly sampled a mask size from $[0,35)$ for every training data. For pitch/formant shift, we sampled a pitch multiplier from $(\frac{1}{1.2},1.2)$ and a formant multiplier from $(\frac{1}{1.1},1.1)$.

First, we compare the proposed method with other baselines using the conventional evaluation metrics. The methods proposed by Frank el al. [14] and Kreuk et al. [7] are working in the text-independent scenario, and Lin et al. [16] performs forced alignment given a phoneme sequence. These baselines are all optimized under the supervised learning setting. We also trained and evaluated a non-autoregressive (non-AR) version of SuperSeg with boundary embedder excluded. In Table 1, we report the results of precision (P), recall (R), F1-score (F1), and R-value (R-val) with a tolerance level of 20 milliseconds. The proposed model, SuperSeg (AR), achieves higher scores of F1-score and R-value than all the previous models in both TIMIT and Buckeye benchmarks. It’s noteworthy that our text-independent model trained solely on the TIMIT or Buckeye corpus outperforms the previous state-of-the-art text-dependent model [16] that leverages the pretrained wav2vec 2.0 model. The unexpected result is that, however, the non-AR model of SuperSeg shows better performance than the proposed AR model. In section 3.4, we point out that the conventional evaluation criteria are vulnerable to the multiple but redundant contributions of adjacent boundaries for scores. To verify this, we investigated the boundaries predicted by the AR and non-AR SuperSeg models and the results are shown in Fig. 3. It can be observed that the non-AR model produces some repeated boundaries around the target boundaries to exploit the vulnerability of the evaluation metrics. On the other hand, the AR model detects phoneme boundaries appropriately without duplicated predictions.

Table 2 shows the results of several models using the proposed algorithm for computing precision and recall. First of all, the F1-score and R-value are all dropped compared to the scores in Table 1. This is a preordained outcome since the proposed algorithm does not allow multiple commitments of each boundary to the evaluation scores. Secondly, the scores of the non-AR model are significantly reduced. This demonstrates that the non-AR model exploits the weaknesses of the conventional metrics and actually does not operate as desired. The other baseline [7] also shows degraded performance on the proposed metrics. Thirdly, the proposed AR model shows decent performance comparable to the previous results with little decrease in the scores. Through this, we can verify that the proposed AR model produces trustworthy phoneme-level segments that fit our expectation.

To check the effect of frequency masking and pitch/formant perturbation, we additionally trained three SuperSeg models with different data augmentation settings. The experimental results are presented in Table 3. When trained without data augmentation, the model performance becomes slightly degraded in both TIMIT and Buckeye corpora. Interestingly, when one of the data augmentations is used, the F1-score and R-value are comparable to or even better than the scores of the model trained with both techniques. This suggests that the data augmentation is useful for the training of SuperSeg to a certain extent.