Stutter-Solver: End-to-end Multi-lingual Dysfluency Detection

We first run MFA to align raw VCTK speech with its ground truth text, obtaining 50 Hz phoneme-level force alignment that matches the EMA features from the AAI module. Various types of dysfluency are then introduced by editing the EMA features: Repetition: The target phoneme segment is duplicated 2-4 times. Replace: We sample a random phoneme from the current EMA feature to replace the target phoneme. Block: A silence frame with 10-15 units is inserted after the target phoneme, with the silence frames sampled from the beginning of the current EMA feature. Missing: The target phoneme is removed. Prolongation: Interpolating within the target phoneme, extending its duration by 4 to 6 times its original length. For repetition and prolongation, the target phonemes are respectively the first phoneme of a randomly selected word and a randomly chosen vowel. For other dysfluency types, the target phonemes are selected arbitrarily without any other restrictions. To ensure smooth auditory perception, we insert a 2-unit interpolate buffer frame before and after each modification. All the interpolation operations mentioned above use bilinear interpolation. Besides phoneme-level modifications above, we also implemented word-level repetition and missing, where the target word is modified instead of the target phoneme, with all other aspects remaining identical. The whole pipeline is depicted in Fig. 2.

Note that only the English version of articulatory-encodec model is available at this time, limiting our simulation contribution to English. However, we explored multi-lingual simulation in TTS-based Simulation, detailed in the next section.

The pipeline of TTS-based simulation can be divided into following steps: 1) Dysfluency injection: for VCTK-Pro, we convert VCTK [22] text into IPA sequences via the VITS phonemizer, and for AISHELL3-Pro, we convert Mandarin text into pinyin sequences. We then add different types of dysfluencies at the phoneme/word(English) and pinyin (Chinese) level according to the TTS rules (Sec. 3.2). 2) VITS [21] inference: We take dysfluency-injected IPA/Pinyin sequences as inputs, conduct the VITS inference procedure and obtain the dysfluent speech. 3) Annotation: We retrieve phoneme alignments from VITS duration model, annotate the type of dysfluency on the dysfluent region.

For VCTK-Pro, we incorporate phoneme and word-level dysfluency; for AISHELL3-Pro, we introduce character-level dysfluency. Dysfluencies are simulated via TTS rules [19], with examples provided in Fig. 3.

In VCTK-Pro, we introduce co-dysfluency, adding multiple dysfluencies into a single utterance. Co-dysfluency is categorized into single-type and multi-type. For single-type, we insert 2-3 instances of the same type of dysfluency (involves every type mentioned above) at various positions within an utterance. For multi-type, we incorporate 5 combinations of dysfluencies: (rep-missing), (rep-block), (missing-block), (replace-block) and (prolong-block), with 2 random positions chosen for each combination within the utterance. Note that due to ethic concerns, de-identification techniques [25] might also be involved in the process.

The statistics of three simulated datasets are detailed in Table. 1.

We obtain $|c_{text}|\times|z|$ monotonic attention matrix $A$ from VITS [21] that represents how each input phoneme aligns with target speech, where $c_{text}$ is text dimension and $z$ the speech duration. For training, we use the soft alignments $A$ and apply a softmax operation across the text dimension, computing the maximum attention value for each time step. To calculate the soft alignments, we use the original pre-trained text and speech encoders. The former is a Transformer encoder [26] with relative positional embeddings, and the latter is a model that uses non-causal residual blocks used in WaveGlow [27]. This soft-alignment attention matrix is then passed into the detection head for training and inference.

We adopt the same spatial-temporal encoders as  [19]. It consists of a region-wise spatial encoder and a temporal encoder.

The speech utterance is split into segments of fixed steps. For each segment, we are going to predict three things: (1) the dysfluency confidence score (if and how confident we are that dysfluency exists in this segment), denoted as $y_{i}$, (2) the boundary of the dysfluencies $b_{\text{start}}$ and $b_{\text{end}}$, and (3) the dysfluency type $c_{n}$: whether that dysfluency is a block, repetition, replacement, insertion, or missing word. The bound values are normalized between 0-1 using fixed padded lengths as the max bound values. The balancing factors are $\lambda_{\text{bound}}=5$, $\lambda_{\text{conf}}=1$, and $\lambda_{\text{class}}=0.5$. $S$ is the number of regions and $n$ is the number of classes. $\mathbbm{1}_{obj}$ indicates whether the dysfluency appears in that segment. The loss function is denoted by the following equation:

• •

  VCTK [22] includes recordings from 109 native English speakers. Each speaker reads out about 400 sentences from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive. This corpus encompasses about 48 hours of accented speech. It is used for simulating VCTK-Pro and VCTK-Art.

• •

  AISHELL-3 [23] is a large-scale and high-fidelity multi-speaker Mandarin speech corpus which includes 218 native Chinese mandarin speakers with roughly 85 hours of emotion-neutral recordings. It is used for simualting AISHELL3-Pro.

• •

  LibriStutter [14] is a synthesized dataset which contains artificially stuttered speech and stutter classification labels for 5 stutter types. It was generated using 20 hours of audio selected from the ‘dev-clean-100’ section of [29].

• •

  UCLASS [30] contains recordings from 128 children and adults who stutter. Only 25 files have been annotated and did not annotate for the block class, we only used those files and did not use the block class for subsequent datasets.

• •

  SEP-28K is curated by [31], contains 28,177 clips extracted from publicly available podcasts. These clips are labeled with five event types including block, prolongation, sound / word repetition and interjection. Clips labeled as “unsure" in the were excluded from the dataset.

• •

  Aphasia Speech is collected from our clinical collaborators, our dysfluent data comprises 46 participants (38 English speakers and 8 Chinese mandarin speakers) diagnosed with Primary Progressive Aphasia (PPA), larger than the data used in  [1, 2] which only has 3 English speakers.

We trained the detector for 30 epochs using a 90/10 train/test split, which was separately applied to three simulated datasets. We utilized a batch size of 64 and leveraged the Adam optimizer, configured with beta values of 0.9 and 0.999 and a learning rate of 3e-4. We choose not to use dropout or weight-decay in our training process. Training on VCTK-Art, VCTK-Pro (without co-dysfluency) and AISHELL3-Pro requires a total of 39, 41 and 36 hours respectively, on a RTX A6000.

• •

  Phoneme Error Rate (PER) is a measure of how many errors (inserted, deleted, and changed phonemes) are predicting phoneme sequences compared to the actual phoneme sequence. It calculated by dividing the number of phoneme errors by the total number of phonemes.

• •

  Accuracy (Acc.) refers to the correctness of predictions regarding types of dysfluency within regions that exhibit some form of dysfluency.

• •

  Bound loss is calculated as the mean squared error between the predicted and actual boundaries of dysfluent regions within a 1024-length padded spectrogram, which is then converted to a time scale using a 20ms sampling frequency. For co-dysfluency analyses, the bound loss is averaged across all identified dysfluent regions.

• •

  Time F1 [1] measures the accuracy of boundary predictions by assessing the overlap between predicted and actual dysfluent region bounds. A sample is classified as a True Positive if any intersection occurs between these bounds.

To assess the performance of trained detector, we conduct evaluations on three simulated datasets, as well as on the PPA data. The results, which include type-specific detection accuracy and bound loss metrics, are detailed in Table. 4 for the simulated datasets and in Table. 5 for the PPA data. Additionally, we compared our results with previous works by validating it on UCLASS, Libristutter, and SEP-28K, where we computed type-specific accuracy and Time F1, as shown in Table. 6.

In Table.4, we used H-UDM [2] as the baseline. Both versions of Stutter-Solver(VCTK-Art and VCTK-Pro) surpassed H-UDM across all metrics. Notably, Stutter-Solver(VCTK-Pro) showed stronger results for English. Additionally, AISHELL3-Pro performed even better, likely due to the unique pronunciation traits of Chinese and its noticeable character-level dysfluency. In Table. 6, we presented our results using publicly available datasets (UCLASS, LibriStutter, and SEP-28K). Since the original benchmarks use private test sets, direct accuracy comparisons may not be completely fair. We instead emphasized time-aware detection, reporting the Time F1 score for each dataset. All baselines, except H-UDM, scored 0. Our proposed methods consistently outperformed H-UDM in these evaluations. In Table. 5, both versions of Stutter-Solver outperformed H-UDM, and the Chinese model performed best on Chinese PPA. However, the average accuracy remained low, underscoring the challenge of accurately capturing the real distribution of dysfluency.

In Section 3.2, we incorporated co-dysfluency into VCTK-Pro. We trained Stutter-Solver on single-type, multi-type, and mixed-type (single & multi) co-dysfluency respectively, and measured average accuracy and bound loss using corresponding simulated data. The results, shown in Table. 7, demonstrate that our detector’s performance on co-dysfluency matches its capability in simpler scenarios with only one dysfluency per utterance. This indicates that our detector handles co-dysfluency effectively. It is worth noting that this fundamental property is missing in previous work.

In addition to training the detector separately on single languages, we trained it simultaneously on two languages to evaluate its performance in a multi-lingual scenario. We randomly sampled 300 hours of data from both VCTK-Pro and AISHELL3-Pro for training. The results, presented in Table.  8, show that multi-lingual training slightly improved detection performance for English but significantly reduced it for Chinese compared with training separately on a single language. This indicates that multi-lingual training has varying effects on detection accuracy depending on the language. It is important to note that our method does not require additional language-specific dysfluency templates, in contrast to the previous state-of-the-art work by [2].