Including Facial Expressions in Contextual Embeddings for Sign Language Generation

Sign languages are spatiotemporal languages and are articulated by using the hands, face, and other parts of the body, which need to be visible. In contrast to spoken languages which are oral-aural languages, sign languages are articulated in front of the top half of the body and around the head. No universal method such as the International Phonetic Alphabet (IPA) exists to capture the complexity of signs. Gloss annotation is often used to represent the meaning of signs in written form. Glosses do not provide any information about the execution of the sign, only about its meaning. Even more, as glosses use written languages rather than the sign language, they are a mere approximation of the sign’s meaning, representing only one possible transcription. For that reason, glosses do not always represent the full meaning of signs as shown in Figure 1.

Every sign can be broken into four manual characteristics: shape, location, movement, and orientation. Non-manual components such as mouth movements (mouthing), facial expressions, and body movements are other aspects of sign language phonology. In contrast to spoken languages, signing occurs simultaneously while vowels and consonants occur sequentially. Although the vocabulary size of ASL in dictionaries is around 15,000 Spread the Sign (2017) compared to approximately 170,000 in spoken English, the simultaneity of phonological components allows for a wide range of signs to describe slight differences of the same gloss.

While in English various words describe largeness (big, large, huge, humongous, etc.) in ASL, there is one main sign for “large”: big. However, through modifications of facial expressions, mouthing, and the size of the sign, different levels of largeness can be expressed just as in a spoken language Grushkin (2017). To communicate spoken concepts without a corresponding sign fingerspelling—a manual alphabet—is sometimes used. Baker et al. (2016)

Facial expressions are grammatical components of sign languages that encode semantic representations, which, when excluded leads to loss of meaning. Facial expressions in particular have an important role in distinguishing different types of sentences such as WH-questions, Yes/No questions, doubt, negations, affirmatives, conditional clauses, focus and relative clauses da Silva et al. (2020). The following example shows how the same gloss order can present a question or an affirmation Baker et al. (2016):

In this example, what makes sentence b) a question are raised eyebrows and a forward and/or downward movement of the head/chin in parallel to the manual signs.

In addition, facial expressions can differentiate the meaning of a sign assuming the role of a quantifier. Figure 1 shows different signs for the same gloss, regen (rain). We can observe from the text transcript (in blue) that the news anchor says “rain” in the upper example but “heavy rain” in the lower. This example shows how gloss annotations are not perfect transcriptions of sign languages as they only convey the meaning of manual aspect of the signs. Information conveyed through facial expressions to show intensities are not represented in gloss annotation. To view the loss of information that occurs in gloss annotation we used Spacy Honnibal and Montani (2017) to compute the Part-of-Speech (POS) annotation for text and gloss. In Table 1 the occurrence of nouns, verbs, adverbs, and adjectives are shown for text and gloss over the entire dataset. We can see that although gloss annotations have lower occurrence for all POS, the difference is statistically significant for adjectives with $p<0.05$. To calculate this significance, we performed hypothesis testing with two proportions by computing the Z score. We used t-tests to determine statistical significance of our model’s performance.

Several advances in generating sign poses from text have been recently achieved in SLG, however there is limited work that considers the loss of semantic information when using gloss to generate poses and aligned facial expressions. Previous work has generated poses by translating text-to-gloss (T2G) and then gloss-to-pose (G2S) or by using either text or gloss as input Stoll et al. (2020); Saunders et al. (2020). We propose a Dual Encoder Transformer for SLG which trains individual encoders for text and gloss, and combines the encoder’s output to capture similarities and differences.

In addition, the majority of previous work on SLG has focused mainly on manual signs Stoll et al. (2020); Saunders et al. (2020); Zelinka and Kanis (2020); Saunders et al. (2021b). Saunders et al. (2021a) are the first to generate facial expressions and mouthing together with hand poses. The representation used for the non-manual channels is the same as for the hand gestures, namely coordinates of facial landmarks. In this work we explore the use of facial Action Units (AUs) (see Figure 2) which represent intensities of facial muscle movements Friesen and Ekman (1978). Although AUs have been primarily used in tasks related to emotion recognition Viegas et al. (2018), recent works have shown that AUs help detect WH-questions, Y/N questions, and other types of sentences in Brazilian Sign Language da Silva et al. (2020).

As our input sources are words, we need to convert them into numerical representations. Similar to transformers used for text-to-text translations, we use word embeddings based on the vocabulary present in the training set. As we are using two encoders to represent similarities and differences between text and glosses we use one word embedding based on the vocabulary of the text and one using the vocabulary of the glosses. We also experiment by using the text word embedding for both encoders. Given that our target is a sequence of skeleton joint coordinates, facial landmark coordinates, and continuous values of facial AUs with varying length we use counter encoding Saunders et al. (2020). The counter $c$ varies between [0,1] with intervals proportional to the sequence length. It allows the generation of frames without an end token. The target joints are then defined as:

The target joints $m_{t}$ are then passed to a continuous embedding which is a linear layer.

We use two encoders, one for text and one for gloss annotations. Both encoders have the same architecture. They are composed by $L$ layers each with one Multi-head Attention (MHA) and a feed-forward layer. Residual connections He et al. (2016) around each of the two sublayers with subsequent layer normalization Ba et al. (2016). MHA uses multiple projections of scaled dot-products which permits the model to associate each word of the input with each other. The scaled dot-product attention outputs a vector of values, $V$, which is weighted by queries, $Q$, keys, $K$, and dimensionality, $d_{k}$:

Different self-attention heads are used in MHA, which allows to generate parallel mappings of the $Q$, $V$, and $K$ with different learnt parameters.

The outputs of MHA are then fed into a non-linear feed-forward projection. In our case, where we employ two different encoders, their outputs can be formulated as:

with $h_{n}$ being the contextual representation of the source sequence, $N$ the number of words, and $U$ the number of glosses in the source sequence.

As we want to not only use the information encoded in text and gloss, but also their relationship, we combine the output of both encoders with a Hadamard multiplication. As the $N\neq U$, we stack $h_{n}$ vertically for $U$ times and stack $h_{u}$ vertically for $N$ times in order to have two matrices with the same dimensions. Then we multiply both matrices with the Hadamard multiplication. Hadamard multiplication is a concatenation of every element in two matrices, where $a_{i,j}$ and $b_{i,j}$ are multiplied together to get $a_{i,j}b_{i,j}$. This represents concatenating the output vectors from the text encoder with the output of the vectors from the gloss encoder.

Our decoder is based on the progressive transformer decoder (DPT), an auto-regressive model that produces continuous sequences of sign pose and the previously described counter value (Saunders et al., 2020). In addition to producing sign poses and facial landmarks, our decoder also produces 17 facial AUs. The counter-concatenated joint embeddings which include manual and facial features (facial landmarks and AUs), $\hat{j}_{u}$ , are used to represent the sign pose of each frame. Firstly, an initial MHA sub-layer is applied to the joint embeddings, similar to the encoder but with an extra masking operation. The masking of future frames is necessary to prevent the model from attending to future time steps. A further MHA mechanism is then used to map the sym- bolic representations from the encoder to the continuous domain of the decoder. A final feed forward sub-layer follows, with each sub-layer followed by a residual connection and layer normalisation as in the encoder. The output of the progressive decoder can be formulated as:

where $\hat{y}_{u}$ corresponds to the 3D joint positions, facial landmarks, and AUs, representing the produced sign pose of frame $u$ and $\hat{c}_{u}$ is the respective counter value. The decoder learns to generate one frame at a time until the predicted counter value, $\hat{c}_{u}$, reaches 1. The model is trained using the mean squared error (MSE) loss between the predicted sequence, $\hat{y}_{1:U}$ , and the ground truth, $y^{\ast}_{1:U}$ :

We extract three different types of features from the PHOENIX14T dataset: skeleton joint coordinates, facial landmark coordinates, and facial action unit intensities. We use OpenPose Cao et al. (2019) to extract skeleton poses from each frame and use for our experiments the coordinates of 50 joints which represent the upper body, arms, and hands, which we will start referring to as “manual features”. We also use OpenFace Baltrusaitis et al. (2018) to extract 68 facial landmarks as well as 17 facial action units (AUs) shown in Figure 2 to describe “facial features”.

We will compare the performance of our proposed model (TG2S) with two Progressive Transformers Saunders et al. (2020), one using gloss only to produce sign poses (G2S), and one that uses text only (T2S). We train each model only with manual features and also with the combination of manual and facial features through concatenation.

In order to automatically evaluate the performance of our model and the baseline models, we use back translation suggested by Saunders et al. (2020). For that purpose we use the Sign Language Transformer (SLT) Camgoz et al. (2020) which translates sign poses into text and computes BLEU and ROUGE scores between the translated text and the original text. As the original SLT was designed to receive video frames as input, we modified the architecture to enable the processing of skeleton poses and facial features.

Table 2 shows how well the SLT model performs the translation from ground truth sign poses to text when trained and evaluated with the PHOENIX14T dataset. The results show the highest BLEU scores are achieved when training the SLT model only with skeleton joints from hands and upper body, presenting a BLEU-4 score of 11.32 for test set. When facial AUs are added to the hands, body, and face features, the difference from using manual data only is slightly lower, being BLEU-4 of 10.61.

In Table 3 the results of using hands and body joint skeleton as sole input to the baseline models and our proposed model are shown. We can see that our proposed model TG2S shows the highest BLEU-4 scores of 8.19 in test set, compared to 7.84 for G2S and 7.56 for T2S.

Table 4 presents the results of including facial landmarks as well as facial AUs with body and hands skeleton joints as input. Also here we can see that our proposed model outperforms the baseline models showing BLEU-4 score of 5.76 in test set. G2S obtained BLUE-4 score of 6.37 and T2S 5.53.

We see in Tables 3 and  4 that G2S obtained higher scores than T2S. Given that gloss annotations fail to encode the richness of meaning in signs, it appears the smaller vocabulary helps the model achieve higher scores by neglecting information otherwise described in text. Our proposed model is able to obtain better results than G2S by making a compromise of using information from gloss, text, and their similarities and differences. We also can see in both tables that the inclusion of facial information reduces the overall scores. We believe that this might be the case due to the diverse range of facial expressions possible. We cannot directly compare the results of Table 3 and  4 as two different SLT models were used to compute the BLEU scores.

Figure 4 shows the visual quality of our models prediction when using manual and facial information. Both examples show that the predictions captured the hand shape, orientation and movement from ground truth. In the bottom example for rain, the predictions were even able to capture the repetitive hand movement symbolizing falling rain. What can also be noted is that the ground truth is not perfect. In both examples unnatural finger and head postures can be seen. In addition, ground truth is not displaying movements of the eyebrows and mouth in the expected intensities.

Figure 5 shows situations in which the predictions failed to represent the correct phonology of signs. In the first example we see that hand shape, orientation, and position are not correct. The predictions of our models also fail to capture pointing hand shapes as shown in example 2.