PiSLTRc: Position-informed Sign Language Transformer with Content-aware Convolution

In sign videos, we observe that each sign gesture usually lasts about 0.5$\sim$0.6 seconds ($\sim$16 frames). However, the vanilla Sign Language Transformer (SLTR) model aggregates sign features in a frame-wise manner, thus neglecting the local temporal structure of sign gestures. Unlike their work, we develop a content-aware neighborhood gathering method to adaptively select the relevant surrounding features, which are around a specific feature and in a contiguous region. Shown as Figure 3, we obtain the clip-level feature with neighboring features via three steps:

Temporal convolution is a common method to aggregate sequential features. However, convolution layers with a limited receptive field are insufficient to capture the position information [22], which is important for sign gesture understanding. More specifically, the recognition of sign language is sensitive to the frame order. Absolute position encoding used in previous methods [15, 16] is a promising approach to encode position information. However, it is demonstrated direction- and distance-unaware [21]. Inspired by recent work on language modelling [25], we infuse relative position information to the clip-level feature. We first compute the relative position matrix $R\in R^{M\times l_{\_}r}$ between the frame-wise feature and the current feature:

Then we represent the relative position indices in learnable embedding space, and obtain the position embeddings $\Phi_{\_}{rpe}\in R^{M\times l_{\_}r\times d}$. Adding $\Phi_{\_}{rpe}$ to clip-level features $F^{r}$, resulting in position-informed clip-level representation:

Lastly, we aggregate the clip-level features with position information to compressed features $F_{\_}{ag}\in R^{M\times d}$, and apply a residual function:

Similar to the vanilla Transformer model, we feed the aggregated feature $F_{\_}{ag}$ to the self-attention mechanism. Note that, as shown in Figure 2, we only set $K=V=F_{\_}{ag}$, and keep $Q$ as original frame-wise representation $F$. The reason for this design is to maintain the difference within adjacent features in $Q$. Experimental result demonstrates that this network design performs better than $Q=K=V=F_{\_}{ag}$.

Like Camgz et al. [15], we extract frame-wise spatial sign features with CNN backbone from CNN-LSTM-HMM [2]. Then we apply the improved Transformer network to learn SLR and SLT simultaneously. Its setting used in our experiments is based on Camgz et al. [15]. Specifically, we use 512 hidden units, 8 heads, 6 layers, and 0.1 dropout rate.

In our proposed CPTcn model, the size of the select contiguous local similar region $l_{\_}r$ is set to be 16 (about 0.5-0.6 seconds), which is the average time needed for completing a gloss. We analyze the impact of the size in Section IV-C1.

The setting of two temporal convolution layers is F3-S1-P0-F3-S1-P0, where F, S, P denote the kernel filter size, stride, and padding size, respectively. The analysis of different modules of the position-informed convolution is concluded in Section IV-C2.

In the self-attention and cross-attention mechanism, we apply DRPE to inject relative position information. We set the max relative distance $\mathcal{L}$ to be 32 in our experiments. The analysis of the DRPE is conduct in Section IV-C3.

Besides, we train the SLR and SLT simultaneously. Thus we set $\lambda_{\_}R$ and $\lambda_{\_}T$ as the weight of recognition loss and translation loss.

We use the Adam optimizer [40] to optimize our model. We adopt the warmup schedule for learning rate that increases the learning rate from 0 to 6.8e-4 within the first 4000 warmup steps and gradually decay it with respect to the inverse square root of training steps. We train the model on 1 NVIDIA TITAN RTX GPU, and use 5 checkpoints averaging for the final results.

During inference, we adopt CTC beam search decoder with a beam size of 5 for SLR decoding. Meanwhile, we also utilize the beam search with the width of 5 for SLT decoding, and we apply a length penalty [41] with $\alpha$ values ranging from 0 to 2.

In our proposed CPTcn module, we introduce a content-aware neighborhood gathering method to select the relevant surrounding neighbors dynamically. Three potential concerns with using this method are: 1) How many improvements does the content-aware method bring? 2) Must be the selected features contiguous in position? 3) What is the appropriate size of the selected region?

In Table I, we compared three methods to verify the first two questions: the essential of whether the selected region is content-aware and contiguous. For notation, w/o NG means no neighborhood gathering method. Centered NG means directly to select k features centered around the current feature. Sparse NG means dynamically selecting k features with the highest similarity, which may be discontinuous in position. Content-aware NG means to select k contiguous features adaptively based on similarity using our proposed content-aware segmentation method. We can see those neighborhood gathering methods effectively improve the performance. Sparse NG substantially outperforms Centered NG. This gap suggests that the content-aware method is critical for feature selecting. Moreover, Content-aware NG performs better than other methods. This indicates that our content-aware contiguous feature aggregation is more suitable for capturing sign gesture representation.

In Table II, we explore the appropriate size of the select local similar region (LSR). The performance of our model performs best when the size of LSR is 16. This is consistent with the finding that the 16-frame (about 0.5-0.6 seconds) is the average time needed for completing a gloss. Besides, by gathering the larger width regions (for example, 20 frames), we observed slight performance degradation. This is because 20 frames (about 1 second) usually contain more than one gesture and thus lower the performance.

In the first two lines in Table III, we study the relative position encoding in the CPTcn module. Experimental results show that position information is crucial for aggregating the sequential features. Furthermore, compared with absolute position encoding (APE), relative position encoding (RPE) bring improvements with $+1.22$ BLEU scores and $-0.46\%$ WER score on the test dataset. The result supports the conjecture of Yan et. al. [21] that RPE provides direction and distance awareness for sequence modeling compared with APE method.

Moreover, we explore the network design of the temporal feature aggregator method in the $3\sim 4$ lines in Table III. Experiments show that residual connection is essential. And Layer normalization is effective for sequential feature modeling.

We further conduct comparative experiments to analyze the effectiveness of disentangle relative position encoding (DRPE) in the attention mechanism. As shown in Figure 1, we replace absolute position encoding (APE) with DRPE in three places: encoder self-attention, decoder self-attention, and encoder-decoder cross attention. For notation, in Table IV, ”Enc-SA” means self-attention in the encoder module. ”Dec-SA” means self-attention in the decoder module. ”Enc-Dec-CA” means cross attention between encoder and decoder. In the $1\sim 4$ lines of Table IV, we can see that the DRPE method used in the encoder and decoder all brings significant improvements. This further demonstrates that relative position encoding provides the direction and distance awareness for sequence representation learning. In addition, we find that the performance of DRPE used only in the encoder is better than that of DRPE used only in the decoder. This phenomenon suggests that direction and distance information are more critical for sign video learning than sentence representation learning.

As we move to the fourth line in Table IV, the results show that DRPE in encoder-decoder attention also increases the performance. This phenomenon shows that even if the order of the word in the natural language is inconsistent with the sign language gloss, the relative position information still benefits their mapping learning.

Different from DRPE used in DeBERTa  [24], we further explore the effectiveness of different items mentioned in Equation 12 in our task. Experimental results are shown in Table V, the correlations between content and position feature bring significant improvement. Moreover, the position-to-position item also benefits our model. This result is consistent with the conclusion in Ke et al. [26]. Accordingly, we adopt these four items in our disentangled relative position encoding.

In Figure 4, we show two examples with different methods on the SLR task. Equipped with proposed approaches, our PiSLTRc model learns accurate sign gesture recognition and thus achieving significant improvements. Furthermore, we find that the model trained based on CTC loss function tends to predict ”peak” on the continuous gestures. And our proposed CPTcn model is adequate to alleviate this situation. As shown in Figure 4, the recognition of adjacent frames in a contiguous region is more precise.

In Table VI, we show several examples with different models on the SLT task. Compared with the vanilla SLTR model [15], our proposed PiSLTRc produces target sentences with higher quality and accuracy.

Comparing the translation results of the first example as illustrated in Table VI, we see that ”vierzehn (fourteen)” is mistranslated as ”sieben (seven)” in SLRT model. However, it is correctly translated in our PiSLTRc model. As we move to the second example in this table, we see that ”heute nacht (tonight)” is mistranslated as ”morgen (tomorrow)” in SLRT model, and it is correctly in our PiSLTRc model. To sum up, specific numbers and named entities are challenging since there is no grammatical context to distinguish one from another. However, in these two examples, we see that our model translates specific numbers and named entities more precisely. This demonstrates that our proposed model has a stronger ability to understand sign videos.

When we move to the third and fourth example in the Table VI, we see that our model generate complete sentence with less under-translation. For example, in the third example, ”gebietsweise länger andauernder regen (rain lasting longer in some areas)” is under-translated in SLTR model, while it is correctly translated as ”bis zum teil dauerregen (partly constant rain)” in our PiSLTRc model.

In summary, our proposed model performs better than the previous SLTR model when facing the specifical numbers and name entities, which are challenging to translate since there is no grammatical context to distinguish one from another. Moreover, the sentences produced follow standard grammar. Nevertheless, it may be improved on the translation quality of the long sentences in the future.

We leverage neighboring similar features to enhance sign representation. The selected features are in a fixed-size region. This is not consistent with the characteristics of sign language. That is to say, the number of frames corresponding to different sign gestures is dynamic.

We evaluate this task in three datasets: PHOENIX-2014-T, PHOENIX-2014 and CSL.

In Table VII, we compare our model with several methods for the sign2gloss task on PHOENIX-2014-T dataset. DNF [6] adopt iterative optimization approaches to tackle the weakly supervised problem. They first train an end-to-end recognition model for alignment proposal, and then use the alignment proposal to tune the feature extractor. CNN-LSTM-HMM [2] embeds powerful CNN-LSTM models in multi-stream HMMs and combines them with intermediate synchronization constraints among multiple streams. Vanilla SLTR-R [15] uses the backbone pretrained with CNN-LSTM-HMM setup and then employes a two-layered transformer encoder model. FCN [10] is built upon an end-to-end fully convolutional neural network for cSLR. Furthermore, they introduce a Gloss Feature Enhancement (GFE) to enhance the frame-wise representation, where GFE is trained to provide a set of alignment proposals for the frame feature extractor. STMC (RGB) [12] proposes a spatial-temporal multi-cue network to learn the video-based sequence. For a fair comparison, we only selected the RGB-based model of STMC without leveraging the additional information of hand, face, and body pose. PiSLTRc-R is our model which is trained when the weight of translation loss $\lambda_{\_}T$ is set zero. Similar to vanilla SLTR-R, our work extracts feature from the CNN-LSTM-HMM backbone. As shown in this table, our proposed PiSLTRc-R surpasses the vanilla SLTR model by $12\%$ and $7\%$ on Dev and Test datasets, respectively. Furthermore, in the RGB-based models, we achieve state-of-the-art performance on the sign2gloss task.

In Table VIII we also evaluate our PiSLTRc-R model on the PHOENIX-2014 dataset. Compared with existed baseline models, our proposed model achieves comparable results. Note that the vanilla SLTR-R does not report the experimental results on the PHOENIX-2014 dataset. We implement it by ourselves. Compared with SLTR-R, our PiSLTRc-R model gains $4\%$ and $5\%$ improvements on Dev and Test datasets, respectively.

In Table XI we conduct experiments on CSL dataset. We see that our proposed PiSLTRc-R model achieves state-of-the-art performance. Compared with the SLTR-R model, our PiSLTRc-R model gains $24\%$ improvements on the Test datasets (5,000 examples split by ourselves), respectively.

In Table IX, we compare our approach with several sign2text methods on PHOENIX-2014-T dataset. The RNN-based model [14] adopt full frame features from Re-sign. TSPnet [16] utilizes I3D [51] to extract the spatial features, and further finetune I3D on two WSLR datasets [52, 53]. Multi-channel [17] allows both the inter and intra contextual relationship between different asynchronous channels to be modelled within the transformer network itself. PiSLTRc-T is our model that training with the weight of recognition loss $\lambda_{\_}R$ being zero. Like in sign2gloss, SLTR-T and our PiSLTRc-T model utilize the pretrained feature from CNN-LSTM-HMM. Experimental results show that our proposed model achieves state-of-the-art performance and surpasses the vanilla SLTR-T model by $3.8\%$ and $5.6\%$ BLEU-4 scores.

In Table X, we compare our model on sign2(gloss+text) task. In this task, we jointly learn sign language recognition and translation simultaneously. Namely, $\lambda_{\_}R$ and $\lambda_{\_}T$ are set as non-zero. Note that different settings will obtain different results. Weighing up the performance on recognition and translation in our experiments, we set $\lambda_{\_}R=\lambda_{\_}T=1.0$. Compared with vanilla SLTR, our model gains significant improvements on both two tasks. Experiments demonstrate that our proposed techniques bring significant improvements for recognition and translation quality based on the sign language Transformer model.