Articulatory Representation Learning Via Joint Factor Analysis and Neural Matrix Factorization

Given vocal tract video representation $X=[X_{1},X_{2},...,X_{t}]\in\mathbb{R}^{2p\times t}$ which is a set of consecutive $t$ frames of vocal tract contours, guided factor analysis [14, 13] aims to decompose $X$ into articulator-specific factors $F=[F_{1},F_{2},...,F_{q}]\in\mathbb{R}^{2p\times q}$ and time domain representation factor scores $Y=[Y_{1},Y_{2},...,Y_{q}]\in\mathbb{R}^{t\times q}$, where $2p$ indicates x,y coordinates of contour vertices and $q$ denotes number of factors. Guilded factor analysis can be formulated in Eq. 1.

$$X=FY^{T}=\Sigma_{i=1}^{q}F_{i}Y_{i}^{T}$$

(1)

The objective of guilded factor analysis is to parameterize the vocal tract representation $X$ into the linear combination [13] of $q$ factors $F_{1},F_{2},...,F_{q}$ such that each factor characterizes spatial variation in the position and shape of an articulator. The factor scores $Y$ characterizes the temporal variation in the position and shapes of the articulators i.e. factors. In this work, we consider 5 types of articulators: jaw, tongue, lip, velum and larynx. We have one factor for each articulator. Thus, there are 5 factors in total:

$$F=[F_{jaw},F_{tongue},F_{lip},F_{velum},F_{larynx}]$$

(2)

Factors are extracted from the vocal tract representation via eigenvalue decomposition algorithm. In order to obtain the articulator-specific representations, we apply a articulator-specific projection matrix on $X$ such that only the contour points on the current articulator are kept while other entries are set to zero. The projection mechanism can be formulated in Eq. 3, where $art\in\{jaw,tongue,lip,velum,larynx\}$. Denote a binary and diagonal projection matrix as $P_{art}\in\{0,1\}^{2p\times 2p}$. Given an index $i\in[1,p]$, the entry $P(2i,2i)$ and $P(2i+1,2i+1)$ which correspond to $x$ and $y$ component of $i$th contour point are 1 if and only if this point is on the current articulator $art$. $P_{art}$ is manually derived via the articulatory segmentation labels of the data [13].

$$X_{art}=P_{art}^{T}X$$

(3)

It is also possible to keep a set of articulators since the motion of some articulators such as jaw, tongue, lip co-occur with each other. For example, if we are going to keep both jaw and lip parts, the joint projection will happen: $X_{jaw,lip}=(P_{jaw}^{T}+P_{lip}^{T})X$. In the next step, eigenvalue decomposition is separately applied to extract the articulator-specific factors. Consistent with [13], factors are extracted in a two-step manner.

First, we only extract jaw factors which capture the most part of vocal tract motions. Following [13], the jaw factors $F_{jaw}\in\mathbb{R}^{2p}$ are defined in Eq. 4, where $union=\{jaw,tongue,lip\}$. $Q$ and $\Lambda$ denote the eigenvector matrix and diagonal variances matrix for the covariance matrix $X^{T}X$. For example, $Q_{union}\Lambda_{union}Q_{union}^{-1}=X_{union}^{T}X_{union}$. Different from [13], we do not apply a scaling factor for convariance matrix since it has little effect to the results. Note that $Q_{jaw}$ and $\Lambda_{jaw}$ capture the direction and variance of jaw movement. Multiplying them with $Q_{union}\Lambda_{union}Q_{union}$ will make the jaw factors capture the joint motion from jaw, lip and tongue articulators. More details can be checked in [13]. After obtaining jaw factor $F_{jaw}$, the vocal tract data can be recovered via the Eq. 5, where $F^{+}$ denotes Moore-Penrose pseudo inverse.

$$F_{jaw}=Q_{union}\Lambda_{union}Q_{union}^{-1}Q_{jaw}\Lambda_{jaw}^{-\frac{1}{2}}$$

(4)

$$\hat{X}=XF_{jaw}F_{jaw}^{+}$$

(5)

Second, we extract the tongue, lip, velum and larynx factors. The motion captured by these factors should be independent of the jaw factors [13]. In order to remove such dependence, we should remove the jaw factor component from the vocal tract data via the Eq. 6. In the next step, we still apply the same mask as Eq. 3 to obtain the articulatory-specific jaw-free vocal tract contours, which is formulated in Eq. 7.

$$X^{other}=X(I-F_{jaw}F_{jaw}^{+})$$

(6)

$$X^{other}_{art}=P_{art}^{T}X^{other}$$

(7)

Following [13], the factors for the a certain articulator $art\in\{tongue,lip,velum,larynx\}$ can be derived via Eq. 8, where $Q^{other}_{art}\Lambda^{other}_{art}(Q^{other}_{art})^{-1}=(X^{other}_{art})^{T}X^{other}_{art}$

$$F_{art}=Q^{other}_{art}\Lambda^{other}_{art}(Q^{other}_{art})^{-1}Q_{art}\Lambda_{art}^{-\frac{1}{2}}$$

(8)

For all of factors, we just take the first principle component.

Once the factors for jaw, tongue, lip, velum and larynx articulators have been finalized, the factor scores can be derived via the simple matrix inversion process of Eq.1. This is shown in Eq. 9, where $F^{+}$ denotes Moore-Penrose pseudo inverse.

$$Y=X^{T}F^{+}=X^{T}[F_{jaw},F_{tongue},F_{lip},F_{velum},F_{larynx}]^{+}$$

(9)

The rtMRI dataset includes eight (four male, four female) speakers of American English [15]. All the speaker were asked to read the same visually presented text from a paper card. The speakers produced the sequence of utterances ten times. The vocal tract constrictions were recorded in the real-time MRI videos. The real-time MRI pulse sequence parameters can be checked in [13]. The video resolution is 83 frames per second. We directly use the segmentation results from [13] which adopts the span segmentation methods [18]. 5 articulators are involved as mentioned in Sec. 2.1 and they are: jaw, tongue, lips, velum and larynx. The total number of points for each image is $2p=400$. For phoneme experiments, we also extract the mel-spectrogram(win=1024,hop=256) and WavLM [19] base features from the audio waveform. Sampling rate for rtMRI audios is 16k. We use MFA [20] to extract monophones given text-audio pairs and there are in total 72 monophones.

The entire articulatory representation learning pipeline is shown in Fig.1. Guided factor anaysis is done offline and we use the same implementation as [13]. The encoder takes factor scores and generates the gestural scores, which is fed into decoder to reconstruct the factor scores. We use similar encoder/decoder as [11]. The encoder consists of 2 convolutional layers and the decoder is a single convolutional layer. Denote $(\cdot,\cdot)$ as $(out\_channels,kernel\_size)$. The configurations for these three layers are $(64,3),(D,3),(21,2p)$ respectively, where the number of points in a image $2p=400$, number of gestures $D=15$ and window size $K=21$ for convolutive matrix factorization. Note that the weights of the decoder are the ”gestures” that is defined in [11]. In this work, the gesture is the multiplication of ”gestures” and factors, as shown in Eq. 11. The phoneme recognizer takes different types of features as input and is optimized via CTC [17] training. For rtMRI data, we simply reshape each frame of image into a hyper-vector so that the entire rtMRI video sequences become a 2D feature. For all the other features, we just keep their original forms. There are two types of phoneme recognizes: base and large. The base model consists of 3-layer multi head attention blocks, where the number of attention heads are 4 and feed-forward layer dimension is 128. The large model consists of 6-layer multi head attention blocks, where the number of attention heads are 8 and feed-forward layer dimension is 256.

We use the same way as [13] to first obtain factors and factor scores. In the next step, we consider two sets of experiments. (1) We perform rtMRI factor scores resynthesis to extract the gestural scores and to derive the gestures. (2) We use various audio features including rtMRI original data, factor scores, gestural scores, mel spectrograms and WavLM [19] features to perform phoneme recognition. For gestural scores, we also furthermore fine-tune both encoder and decoder, which is denoted as Gestural Scores(FT) in Table. 1. For the overall loss function defined in Eq.13, we set $\lambda_{1}=0.5$. By default $\lambda_{2}=0$ unless we finetune the gestural score where $\lambda_{2}=0.3$. For all experiments, adam [21] optimizer is used with an intial learning rate of 0.001 and weight decay of 4e-4. All the experiments are trained for 1k updates with a batch size of 4. The learning rate is decayed every 10 updates with a factor of 0.95. For phoneme recognition experiments, we also vary the training size in a manner that we gradually increase the number of speakers. For each speaker, we have 3 random utterances as test set and the remaining as training set. Beam search with a width of 10 is used for decoding. We evaluate the intelligibility, generaliability and efficency of the gestural scores and the interpretability of both gestural scores and gestures.