Articulation GAN: Unsupervised modeling of articulatory learning

Speech synthesis from articulatory representations has recently been performed using deep neural networks [10, 11, 12, 13, 14, 9]. The objective in most existing proposals, however, is to synthesize waveforms from articulatory representations in a supervised setting, rather than a fully unsupervised generation of the articulatory representations themselves. [15, 16] proposes an autoencoder model that learns to encode and decode between motor parameters and auditory representations in an unsupervised manner. However, this model trains both the encoding and decoding aspects of the model simultaneously, and focuses on the relationship between auditory representations and a motor latent space. By contrast, our GAN model is trained with a static pretrained articulatory model similar to how children learn to speak with a full set of articulators. In addition, rather than decoding back and forth between motor and auditory information, our model is able to generate articulatory parameters directly by sampling from a general-purpose latent space. To our knowledge, this paper presents the first architecture in which a generative model learns to produce unprompted articulatory gestures that result in speech in a fully unsupervised way.

Computational models of language almost always disregard the articulatory component. Currently, articulatory phonology is a proposal that comes closest to modeling linguistic representations from articulatory representations [17, 18], and phonological structure can be inferred from articulatory data [19]. However, these models take articulatory gestures as a given (as measured on human subjects) and do not model unsupervised learning and generation of articulatory gestures from auditory feedback.

A model of unsupervised articulatory learning is not only a more realistic representation of human speech, but is useful for conducting cognitive simulations that have the potential to reveal which properties of speech emerge because of articulatory factors and which properties are cognitively conditioned [20]. In engineering application, learning to generate plausible articulatory gestures with accompanied synthesized speech is useful for lip synchronization [21] (with potential applications in robotics or gaming industry). The modelling of articulatory information has also been identified as being useful in the detection of audio deepfakes [22]. Generation of articulatory gestures is thus potentially useful for creating more realistic speech synthesis technologies, as well as providing another adversarial approach that deepfake detectors can use to improve their accuracy.

The Articulatory Generator $G$ is adapted from the Generator network from WaveGAN [5]. It takes as input a latent noise vector $z$ and uses 5 layers one-dimensional transpose convolutions to upsample the noise into waveform data $G(z)$. Unlike WaveGAN, our Articulatory Generator generates 13 channels (one channel of voicing plus the x- and y-axis for 6 articulators). Due to the physical model's low sample rate, the dimensionality of each layer is also lower than in WaveGAN, with individual dimensionalities of $32\times 512,64\times 512,128\times 256,128\times 256,256\times 13$, respectively.

We take the EMA-to-speech encoder trained on MNGU0 from [9] to be a physical model of articulation $\mathcal{A}$. This autoregressive model takes as input 13 channels of time-series and outputs a 16 kHz waveform corresponding to speech. Specifically, the 13 channels of articulatory features include one channel of voicing, plus the $x$ and $y$ coordinate positions each of the lower incisor, upper lip, lower lip, tongue tip, tongue body, and tongue dorsum.

To test how well the ArticulationGAN performs compared to WaveGAN, we randomly generated 200 outputs from the ArticulationGAN and WaveGAN models (400 total). A trained phonetician who is not a coauthor was hired to annotate and transcribe the outputs in order to avoid potential bias and to account for noise in the outputs. The outputs were annotated as (i) intelligible words of English, (ii) intelligible sequences of sounds that are not words of English, and (iii) unintelligible outputs. The results are given in Figure 1. Intelligible outputs include all sounds that were transcribable by the trained phonetician; the proportion of comfortably intelligible outputs is likely lower. The WaveGAN model performs slightly better on the intelligibility task (87% vs. 72%), but the ArticulationGAN outputs a higher proportion of intelligible outputs (words and non-words) that are not part of training data (innovative outputs).

The weaker intelligibility of ArticulationGAN is likely due to its more difficult training objective. The WaveGAN Generator only needs to produce outputs that are themselves similar to the training distribution drawn from TIMIT. On the other hand, the Articulatory Generator must produce outputs that approximate the TIMIT data after being passed through a fixed articulatory model that has been trained on MNGU0 data. As previously mentioned, TIMIT data is drawn from 630 speakers across eight dialects of American English [24], while MNGU0 data is drawn from a single speaker of British English [26].

Nevertheless, ArticulationGAN produces a higher amount of innovative data compared to both the TIMIT and MNGU0 datasets. The results suggest that the ArticulationGAN not only learns words that are represented in both TIMIT training data and MNGU0 dataset (e.g. wash), but also words that are absent from the MNGU0 dataset and the TIMIT training dataset. For example, the ArticulationGAN generated outputs that were transcribed as wash [”wOS], fast [”fæst], greasy [”gisi], and coffee [”kOfi]. Wash is part of TIMIT and MNGU0 training data. Fast and coffee are only present in the MNGU0 data. Fast is acoustically close to ask in the training data. Coffee is distant to its closest equivalent in the TIMIT training data (greasy), but greasy is absent from MNGU0. The ema2wav model is never trained to generate greasy from EMA, yet our ArticulationGAN generates several outputs that can be reliably transcribed as greasy (Figure 2).

We also observe overrepresentation of w-initial words in the 200 outputs of the ArticulationGAN compared to TIMIT training data ($\text{OR}=1.53,p<0.01$), but not in WaveGAN outputs ($\text{OR}=0.94,p=0.74$). Gestures for [w-] are easier to acquire compared to other initial consonants. It appears that ease of articulation of labial consonants plays a role in articulatory learning in our models (similar to language acquisition [27, 28]).

To analyze unsupervised learning of articulatory gestures in the ArticulationGAN model, we compare real (MNGU0) and generated (ArticulationGAN) EMA channels and corresponding acoustic outputs (waveforms). We analyze articulatory gestures in two generated outputs transcribed as wash [”wOS] and fast [”fæst]. These words were chosen because wash is present in both TIMIT training and MGNU0 data, while fast is an innovative output. Because greasy is fully absent from MNGU0 training data, we cannot compare generated and real EMA for this word.

Figure 3 illustrates the 12 channels plus voicing for wash. We observe the network learns relatively stable articulatory targets, except during transitions between targets or when an articulator does not play an active role for a given phoneme sequence. For example, the x axis of tongue dorsum and the lower incisor position do not play a central role in the articulation of wash, which is why this channel is relatively noisy in Figure 3.

To interpret articulatory gestures and compare real human EMA to generated EMA, we visualize x and y-axis values in 2D space for each electrode placement. Because the Generator has no restrictions that would penalize rapid changes (as is the case in human muscle and movements), we smooth the generated EMA with LOESS smoothing. Figure 4 contains generated and real EMA for wash and an innovative output fast. Tongue tip and lower/upper lip are the most relevant articulators for wash and fast. We observe very similar gestures between GAN and EMA for wash, and an almost identical pattern the lower lip gestures for fast.

We further performed a quantative comparison between gestures for the generated and real EMA. To account for differences in timing, we perform dynamic time warping (DTW) between smoothed generated EMA and real EMA for each dimension (x and y) and each electrode placement. We then compute Pearson's product-moment correlation ($r$) on two time series data for each channel to estimate the time-aligned correlation between generated and real EMA.

The quantitative comparison in Table 2 reveals a high degree of correlation in gestures between real EMA and GAN-generated EMA. Tongue tip gestures in fast are almost identical ($r=0.99$ in x-axis and $r=0.96$ for y axis). We observe that tongue tip, lower lip, and tongue body have highest correlations and the y-axis is better correlated than the x-axis. This is expected as vertical movements are more consequential in these words.

Despite the training complexities discussed in Section 1, the intelligibility of ArticulationGAN's outputs is not substantially lower than that of WaveGAN (Table 1). ArticulationGAN outputs a higher proportion of innovative intelligible outputs. This is not unexpected from cognitive modeling perspective: speech production (articulatory learning) is substantially more difficult than speech perception (acoustic learning), and innovative outputs are common during articulatory learning.

EMA data is a very low-dimensional representation of articulation in human speech. Adding articulatory representations (e.g. more channels or additional articulation data types) might improve performance and provide higher resolution insights about articulation. Also, our model operates with a single Discriminator and a single 5-layer Generator that needs to generate 13 1D channels, which may impact performance. Adding multiple subdiscriminators has also been shown to increase performance in the GAN framework [29].