Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

We build our spatial attention module based on CBAM (Woo et al., 2018) due to its simplicity and effectiveness. As shown in Figure 3(a), we first pick the most informative channel via a channel-wise max pooling (CMP) operation:

where $\mathbf{M}_{1}$ is the squeezed feature map by CMP, and $\mathbf{F}\in\mathbb{R}^{J\times K\times C}$ is the input feature maps.

Besides CMP, CBAM also squeezes the feature maps with an average pooling operation along the channel dimension. However, we propose to dynamically weight the importance of each channel. As shown in Figure 3(a), we first conduct global average pooling (GAP) over $\mathbf{F}$ to gather global spatial information. Then the channel weights $\mathbf{E}\in(0,1)^{1\times 1\times C}$ are simply generated by a channel-wise softmax layer. By a weighted sum along the channel dimension, we can generate another squeezed feature map $\mathbf{M}_{2}$:

Finally, the spatial attention mask $\mathbf{M}$ is generated as:

where $\sigma(\cdot)$ is the sigmoid function, $f_{conv}(\cdot)$ is a 2D-CNN layer with a kernel size of 7×7, and $cat(\cdot,\cdot)$ is a channel-wise concatenation operation. The output feature maps will be a product between $\mathbf{F}$ and $\mathbf{M}$. In this way, important positions can be highlighted while trivial ones can be suppressed.

It should be noted that our channel weights are similar to the channel attention module in CBAM, but ours introduces no extra parameters and can even outperform the vanilla CBAM according to our ablation studies in Table 3.

Simply training the spatial attention module with the backbone may lead to sub-optimal solutions. Given the prior knowledge that signers’ faces and hands are informative regions (IRs), we guide the spatial attention module with keypoints heatmaps extracted by the pretrained HRNet (Sun et al., 2019; Andriluka et al., 2014). Specifically, we first normalize the raw outputs of HRNet linearly to obtain the original heatmaps:

where $\mathbf{I}$ is the raw RGB frame, $f_{H}(\cdot)$ is the pretrained HRNet, and $i\in\{1,2,3\}$ denotes the face, left hand, and right hand, respectively.

There are some defects in the original heatmaps although they can roughly highlight the positions of IRs. As shown in Figure 3(b), some trivial regions, e.g., the top of the face heatmap in the first row and the middle part of the left hand heatmap in the second row, may receive high activation values. Besides, some highlighted regions, e.g., both of the face heatmaps in Figure 3(b), may not cover the IRs entirely. In addition, there is usually a mismatch between the fixed heatmap resolution of the pretrained HRNet and that of the spatial attention masks. Below we will elaborate our heatmap post-processing module that corrects the mismatch.

We first locate the center of each IR from the original heatmaps via a simple argmax operation: $(x_{i},y_{i})=\mathrm{argmax}\ \mathbf{H}_{o}^{i}$. To fit different resolutions of spatial attention masks, we normalize the center as $(\hat{x}_{i},\hat{y}_{i})=(\frac{x_{i}}{H-1},\frac{y_{i}}{W-1})$. Suppose the spatial attention masks have a common resolution of $J\times K$, then a Gaussian-like refined keypoints heatmap is generated for each IR to reduce unwanted noise:

where $0\leq a<J$, $0\leq b<K$. $(\hat{c}_{i}^{x},\hat{c}_{i}^{y})=(\hat{x}_{i}(J-1),\hat{y}_{i}(K-1))$, which denotes the transformed center for each IR under the resolution $J\times K$. $\gamma_{x}$ and $\gamma_{y}$ are two hyper-parameters to control the scale of the highlighted regions. In real practice, we set $\gamma_{x}=\gamma_{y}$. Finally, we merge the three processed IR heatmaps into a single one: $\mathbf{H}_{r}=\underset{i}{\max}\ \mathbf{H}_{r}^{i}\in(0,1)^{J\times K}$.

The spatial attention module is guided by the refined keypoints heatmaps via the SAC loss³³3For implementation, we further compute the average of $\mathcal{L}_{sac}$ over all time steps.:

Within a sign video, each gloss consists of only a few frames. We believe a good SEE for sign languages should take local contexts into consideration. As shown in Figure 4, our SEE is built on QANet (Yu et al., 2018), which consists of a depth-wise temporal convolution network (TCN) layer and a transformer encoder layer. The depth-wise TCN first extracts local contextual information from the frame-level feature sequence, then the transformer encoder models global contexts by its inner self-attention module.

Similar to the class token in BERT (Kenton and Toutanova, 2019), we first prepend a learnable sentence embedding token, [SEN], to the sequential features $\mathbf{s}\in\mathbb{R}^{T\times d}$ defined in Section 3.1:

The input of the SEE is the summation of the feature sequence and the positional embeddings (Vaswani et al., 2017); i.e., $\mathbf{s}^{\prime\prime}=\mathbf{s}^{\prime}+\mathbf{P}$, where $\mathbf{P}\in\mathbb{R}^{(T+1)\times d}$.

Within the SEE, the depth-wise TCN (Wu et al., 2018) layer first models local contexts with a residual shortcut: $\mathbf{s}_{l}^{\prime\prime}=f_{TCN}(\mathbf{s}^{\prime\prime})+\mathbf{s}^{\prime\prime}$. Then the transformer encoder layer gathers information from all time steps to get the sentence embedding:

We can also get the sentence embedding of visual features, $\mathbf{E}_{sen}^{v}$, in the same way.

Directly minimizing the distance between $\mathbf{E}_{sen}^{s}$ and $\mathbf{E}_{sen}^{v}$ will result in trivial solutions. For example, if the parameters of SEE are all zeros, then the outputs of SEE will always be the same. A simple way to address this issue is introducing negative samples. In this work, we follow the common practice (Schroff et al., 2015; Ye et al., 2019; Oord et al., 2018; Hjelm et al., 2019) and sample another video from the mini-batch and take its sequential features as the negative sample. Note that most CSLR models (Min et al., 2021; Hao et al., 2021; Zhou et al., 2020) are trained with a batch size of 2, and our negative sampling strategy will degenerate to swapping under this setting:

where $\mathbf{B}\in\mathbb{R}^{2\times T\times d}$ is a mini-batch of the sequential features, and $neg(\mathbf{B}[\cdot])$ denotes the corresponding negative sample.

We implement SEC loss as a triplet loss (Schroff et al., 2015) and minimize the distances between the sentence embeddings computed from the visual and sequential features of the same sentence, while maximizeing the distances between those from different sentences:

where $d(\mathbf{x}_{1},\mathbf{x}_{2})=1-\frac{\mathbf{x}_{1}\cdot\mathbf{x}_{2}}{\|\mathbf{x}_{1}\|_{2}\cdot\|\mathbf{x}_{2}\|_{2}}$; $\{\mathbf{E}_{sen}^{v},\mathbf{E}_{sen}^{s}\}$ are sentence embeddings of visual and sequential features from the same sentence; $\{\mathbf{E}_{sen}^{v},neg(\mathbf{E}_{sen}^{s})\}$ are sentence embeddings of visual and sequential features from different sentences, and we treat the sentence embedding of the sequential features from a different sentence as the negative sample $neg(\mathbf{E}_{sen}^{s})$; $\alpha$ is the margin.

We first extract signer embeddings to “distill” signer information before dispelling it. A naïve method is simply feeding the frame-level features into an MLP, and treat the outputs of MLP as signer embeddings. In this work, motivated by the superior performance of x-vectors (Snyder et al., 2018) in speaker recognition, we leverage statistics pooling to obtain more robust sentence-level signer embeddings.

Specifically, we first feed the intermediate visual features $\mathbf{F}\in\mathbb{R}^{T\times J\times K\times C}$ into a global average pooling layer to squeeze the spatial dimension and obtain frame-level features⁴⁴4Here we misuse the notation $\mathbf{F}$ in Equation 1. $\mathbf{F}_{s}\in\mathbb{R}^{T\times C}$. Then a statistics pooling (SP) layer is used to aggregate frame-level information:

where $\mathbf{F}_{s}^{mean}\in\mathbb{R}^{C}$ and $\mathbf{F}_{s}^{std}\in\mathbb{R}^{C}$ are the temporal mean and standard deviation of $\mathbf{F}_{s}$, respectively. In this way, $\mathbf{F}_{s}^{SP}$ is capable to capture signer characteristics over the entire video instead of at the frame-level.

After that, a simple two-layer MLP with rectified linear unit (ReLU) function is used to project the statistics into the signer embedding space:

where $\mathbf{W}_{1}\in\mathbb{R}^{C\times 2C},\mathbf{b}_{1}\in\mathbb{R}^{C},\mathbf{W}_{2}\in\mathbb{R}^{C\times C},\mathbf{b}_{2}\in\mathbb{R}^{C}$ represent the two-layer MLP.

Finally, the signer embeddings $\mathbf{E}_{sig}$ are fed into a classifier to yield signer probabilities $\mathbf{p}_{sig}\in(0,1)^{N_{sig}}$, where $N_{sig}$ is the number of signers. The SRM is trained with the signer classification loss, which is simply a cross-entropy loss:

where $i$ is the label of the signer.

If the CSLR backbone is jointly trianed with $\mathcal{L}_{srm}$, it will become the multi-task learning, which, however, cannot promise removing the signer information from the backbone. In this work, we treat each signer as a domain and formulate SI-CSLR as a domain generalization problem in which no test signers are seen during training. The gradient reversal layer was proposed in (Ganin et al., 2016) to address the domain generalization problem by learning features that are discriminative to the main classification task while indiscriminate to the domain gap. More specifically, according to (Ganin et al., 2016), denoting the parameters of the feature extractor, label predictor, and domain classifier as $\theta_{f}$, $\theta_{y}$, and $\theta_{d}$, respectively, the optimization of these parameters can be formulated as:

where $\mathcal{L}_{y}$ and $\mathcal{L}_{d}$ are the main classification and domain classification losses, respectively, $\lambda$ is the loss weight for $\mathcal{L}_{d}$, and $\eta$ is the learning rate.

We adapt Equation 14 by instantiating $\mathcal{L}_{y}$ and $\mathcal{L}_{d}$ as the backbone training loss $\mathcal{L}_{b}$ and signer classification loss $\mathcal{L}_{srm}$, which are illustrated in Figure 5, respectively. We also merge $\theta_{f}$ and $\theta_{y}$ as $\theta_{b}$ to denote the parameters of the backbone, and use $\theta_{s}$ to represent the parameters of the SRM. The new optimization process can be formulated as:

As a result, the SRM itself is trained with $\mathcal{L}_{srm}$ as usual, but the backbone is trained “reversely” so that the extracted features cannot discriminate signers, and the signer information is implicitly removed. We validate the effectiveness of the SRM on two challenging SI-CSLR benchmarks, establishing a strong baseline for future works on SI-CSLR.

We evaluate our method on three signer-dependent datasets (PHOENIX-2014, PHOENIX-2014-T, and CSL-Daily) and two signer-independent datasets (PHOENIX-2014-SI and CSL). The information of these datasets, including language, vocabulary size, train/dev/test splits, number of signers, are available in Table 1. Compared to some widely-adopted datasets in action recognition, e.g., Kinetics-600 (Carreira et al., 2018) with  500K videos and Something-Something v2 (Goyal et al., 2017) with  169K videos, the size of these sign language datasets are quite small. This can also explain why some specific training strategies, e.g., stage optimization and auxiliary training, are suggested necessary for CSLR before.

We use word error rate (WER) to measure the dissimilarity between two sequences.

The official evaluation scripts provided by each dataset are used for measuring the WER.

We first resize the RGB frames to $256\times 256$ and then crop them to $224\times 224$. For the PHOENIX datasets, we adopt stochastic frame dropping (SFD) (Niu and Mak, 2020) with a dropping ratio of 50%. However, due to the longer duration of videos in CSL and CSL-Daily, we implement a seg-and-drop strategy that first segments the videos into short clips consisting of only two frames, and then one frame is randomly dropped from each clip. By doing so, the processed videos retain half of the original frames, while preserving most information. After that, we further randomly drop 40% frames using SFD from these processed videos.

We first choose three representative backbones to validate the effectiveness of our method.

• •

  VGG11+TCN+BiLSTM (VTB). It is widely adopted in some recent works (Zhou et al., 2020; Min et al., 2021). VGG11 (Simonyan and Zisserman, 2015) is used as the visual module, and the sequential module is composed of the TCN and BiLSTM to capture both local and global contexts.

• •

  CNN+TCN (CT). This lightweight backbone only consists of a 9-layer 2D-CNN and a 3-layer TCN, which is proposed in (Cheng et al., 2020).

• •

  VGG11+Local Transformer (VLT). The sequential module is a 2-layer local transformer encoder described in Section 3.6.

To better validate the robustness of our method, we additionally append our local transformer to three mainstream visual backbones including ResNet-18 (He et al., 2016), MobileNet-v3-Small (Howard et al., 2019), and GoogLeNet (Szegedy et al., 2015). To align the channel dimensions of visual and sequential features, we configure the TCN layers in CT and VTB to have an output channel size of 512. Additionally, we set the number of hidden units for the BiLSTM in VTB to $2\times 256$. These adjustments lead to comparable Word Error Rates (WERs) to those reported in the original papers (Cheng et al., 2020; Zhou et al., 2020), maintaining consistency in performance evaluation. We empirically insert the spatial attention module after the 5th CNN layer. For post-processing, we set $\gamma_{x}=\gamma_{y}=14$ based on the experimental results presented in Section 4.3.6. The kernel size of the depth-wise TCN layer in both our SEE and VLT backbone models is set to 5, consistent with (Yu et al., 2018). To determine the margin $\alpha$ in Equation 10, we consider the maximum difference between negative and positive cosine distances and set $\alpha$ to 2. Regarding the signer removal module, we empirically position it after the 5th CNN layer, and the default weight for $\mathcal{L}_{srm}$, $\lambda$, is set to 0.75.

All models are trained using a batch size of 2, following recent works (Hao et al., 2021; Min et al., 2021; Zhou et al., 2020). We employ the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of $1\times 10^{-4}$ and a weight decay factor of $1\times 10^{-4}$. We empirically notice that $\mathcal{L}_{sec}$ decreases at a faster rate compared to $\mathcal{L}_{ctc}$. To ensure consistent training progress between the backbone and SEE, we decrease the learning rate of the SEE by a default factor of 0.1 for the backbone architectures used in our experiments (for CT, we set the factor to 0.01). Following the previous approach (Camgöz et al., 2020), we employ a learning rate scheduling strategy known as plateau. Specifically, if the WER on the dev set does not decrease for 6 consecutive evaluation steps, the learning rate will be reduced by a factor of 0.7. However, since CSL does not have an official dev set, we decrease the learning rate after the 15th and 25th epoch, and subsequently every 5 epochs after the 30th epoch. The total number of training epochs is set to 60.

Following (Niu and Mak, 2020), to match the training condition, we evenly select every $\frac{1}{p_{d}}$-th frame to drop during inference, where $p_{d}$ is the dropping ratio. We adopt the beam search algorithm with a beam size of 10 for decoding.

As shown in Table 2, both SAC and SEC generalize well on different backbones: the performance of all the six backbones can be clearly improved. However, if the spatial attention module is inserted into the backbones without any guidance, i.e., $\text{SAC}^{-}$, the model performance can only be improved slightly, which verifies the effectiveness of $\mathcal{L}_{sac}$. The effectiveness of SEC suggests that explicitly enforcing the consistency between the visual and sequential modules at the sentence level can strengthen the cross-module cooperation, which leads to the performance gain. The improvements due to SAC and SEC are complementary so that using both of them can obtain better results than using only one of them. Besides, since VLT performs the best among the six backbones, we will use it as the default backbone for the following experiments.

Figure 7 shows the visualization results of the learned spatial attention masks of SAC (with $\mathcal{L}_{sac}$) and $\text{SAC}^{-}$ (without $\mathcal{L}_{sac}$) for five test samples. It should be noted that since SAC is deactivated during testing, our comparison is fair. First, it is quite clear that the learned attention masks with the guidance of $\mathcal{L}_{sac}$ look much better. Without the guidance of $\mathcal{L}_{sac}$, the attention masks are quite messy with horizontal lines at the top and many highlights at trivial regions, e.g., the left shoulder of $s_{2}$, the hair of $s_{1}$ and $s_{4}$, and the waist of $s_{3},s_{5}$. This explains why $\text{SAC}^{-}$ can only slightly improve the performance of the backbones as shown in Table 2. Second, our SAC is so robust that the IRs (face and hands) in blurry frames (right columns of $s_{1}$ to $s_{5}$) can still be captured precisely. Third, it is capable of dealing with different hand positions: e.g., both two hands are lower than the face ($s_{1},s_{3}$); one hand is near the face while the other one is not ($s_{1},s_{2},s_{4}$), and hands are overlapped ($s_{5}$).

Within our spatial attention module, each channel can receive a weight to better measure its importance before squeezing the feature maps. Removing the channel weights degenerates to the channel-wise average pooling in CBAM (Woo et al., 2018) and achieves a WER of 21.3%, which leads to a performance drop by 0.5% as shown in Table 3. Although our channel weights share a similar idea with the channel attention module of CBAM, which builds extra linear layers to generate the attention weights, no extra parameters are introduced in our spatial attention module. To further validate their effectiveness, after removing the channel weights, we conduct one more experiment by adding the channel attention module back as CBAM; however, it can only lead to a slight performance gain and cannot outperform ours even with extra parameters.

We discuss in the Section 3.2.3 that the raw heatmaps of HRNet (Sun et al., 2019) consist of too many defects which may hinder the learning of the spatial attention module. As shown in Table 3, the quality of the keypoints heatmaps can make a difference on model performance: directly using the original heatmaps without post-processing achieves a WER of 21.7%, which reduces the performance of SAC by almost 1%.

As shown in the last two rows in Table 3, removing either face or hands region can harm the performance of SAC. The results validate that both signers’ face and hands play a key role in conveying information, which is also mentioned in (Zhou et al., 2020; Koller, 2020).

We think $\gamma_{x}$ and $\gamma_{y}$ are two important hyper-parameters since they control the scale of highlighted regions in keypoints heatmaps. Thus, we conduct experiments to compare the performance of different $\gamma_{x},\gamma_{y}$ as shown in Figure 8. The model performance is worse when they are either too large (cannot cover the informative regions entirely) or too small (cover too many trivial regions). When $\gamma_{x}=\gamma_{y}=14$, the model achieves the best performance.

Our sentence embedding extractor consists of a depth-wise TCN layer and a transformer encoder aiming to model local and global contexts, respectively. As shown in Table 4(a), local contexts are important to sentence embedding extraction as dropping the TCN layer would lead to worse performance. We also compare our method with the common practice, which concatenates the last two hidden states of BiLSTM and treats it as the sentence embedding. Nevertheless, that it underperforms the transformer-based extractors implies the strength of the self-attention mechanism for sentence embedding extraction. Table 4(a) also shows that negative sampling plays a key role in our SEC: without negative sampling, that is, directly minimizing the sentence embedding distance between the visual and sequential features, is not effective.

As shown in Table 4(b), we implement some frame-level constraints to validate the effectiveness of our SEC. First, we replace the sentence embeddings, $\mathbf{v}_{se}$ and $\mathbf{s}_{se}$ in Equation 10, by their corresponding frame-level features to minimize the positive distances while maximizing the negative distances at the frame level. However, it leads to a performance degradation of 0.7% compared to our SEC. We further compare our SEC with VAC (Min et al., 2021), which is composed of two frame-level constraints: visual enhancement (VE) and visual alignment (VA). First, an extra classifier is appended to the visual module to yield frame-level probability distributions (visual distribution). VE is implemented as a CTC loss computed between the visual distribution and the gloss label, which is the same as the one used for training the backbone. Second, VA is simply a KL-divergence loss, which aims to minimize the distance between the visual distribution and the original probability distribution ($p(\phi_{i}|\mathbf{x})$ in Equation 16). Table 4(b) shows that both VE and VA perform much worse than our SEC. The results suggest that our SEC is a more proper way to measure the consistency between the visual and sequential modules.

To verify whether $\mathcal{L}_{sec}$ can really separate positive and negative samples, we provide two examples of video-gloss pairs denoted as $(v_{1},l_{1})$ and $(v_{2},l_{2})$ as shown in Figure 9. The sentence embedding distances of the visual and sequential features of $v_{1}$ and $v_{2}$ are shown in Table 6. It is clear that the distance between the two features of the same video (diagonal entries, positive pairs) can be very small. Otherwise (off-diagonal entries, negative pairs), the distance can be very large (the maximum value is 2.00).

According to (Liu et al., 2018b), the weight for the domain classification loss, i.e., our signer classification loss $\mathcal{L}_{srm}$, is an important hyper-parameter. We fine-tune it from 0 to 1.5 with an interval of 0.25 as shown in Table 6. When $\lambda=0$, the model degenerates to $\text{C}^{2}$SLR and performs worse than other models with $\lambda>0$. The results suggest the importance of removing signer information for SI-CSLR. When $\lambda=0.75$, the model can achieve the best performance with a WER of 33.1%/32.7% on the dev and test set, respectively.

We further conduct ablation studies for the two major components of our SRM, the statistics pooling (SP) and gradient reversal (GR) layer. First, the use of the GR layer decides the type of learning method: feature disentanglement or multi-task learning. As shown in Table 7, it is clear that with the use of GR, models under the feature disentanglement setting can significantly outperform those under the multi-task learning setting. The result implies that removing signer information is effective to SI-CSLR. However, we find that the model, $\text{C}^{2}$SLR+SP, can also outperform the baseline under the multi-task setting. We think this is because the multi-task learning can be seen as a kind of regularization (Zhang and Yang, 2021), which endows the shared networks between the CSLR and signer classification branches with better generalization capability. Similar ideas also appear in some works that jointly train a speech recognition model and a speaker recognition model (Liu et al., 2018a; Pironkov et al., 2016). Finally, the effectiveness of SP also validates that sentence-level signer embeddings are more robust than frame-level ones to achieve signer classification, leading to better performance.

We finally study the effect of the SRM over seen and unseen signers. We first build an extra test set consisting of only seen signers during training by removing videos performed by unseen signers from the official test set of PHOENIX-2014, and then retest “$\text{C}^{2}$SLR” and “$\text{C}^{2}$SLR+SRM” on this extra test set. As shown in Table 8, with a comparable performance over the seen signers, adding the SRM can significantly narrow the performance gap between unseen and seen signers. The results suggest that our SRM can be more helpful for the real-world situation that most testing signers are unseen.

As shown in Table 9, we first evaluate our $\text{C}^{2}$SLR on three signer-dependent benchmarks: PHOENIX-2014, PHOENIX-2014-T, and CSL-Daily.

Our $\text{C}^{2}$SLR follows the idea of auxiliary learning, which also appears in some existing works, e.g., FCN (Cheng et al., 2020) and VAC (Min et al., 2021). FCN proposes a gloss feature enhancement (GFE) module to introduce auxiliary supervision signals into the model training process. However, the GFE module highly relies on pseudo labels (CTC decoded results), which may contain too many errors. Our method only relies on pre-extracted heatmaps, which are quite accurate with the help of our post-processing algorithm, and the model’s inherent consistency: the visual and sequential features represent the same sentence. These two properties enable our method to outperform FCN by more than 3% on both PHOENIX-2014 and PHOENIX-2014-T. Recently, VAC proposes two auxiliary losses at the frame-level, which are not quite appropriate and perform worse than ours according to the comparison in Section 4.3.8. The SOTA work, STMC (Zhou et al., 2020), adopts the complicated stage optimization strategy, which introduces extra hyper-parameters, and needs to manually decide when to switch to a new stage. Our method is totally end-to-end trainable, and it can outperform STMC on both PHOENIX-2014 and PHOENIX-2014-T. To the best of our knowledge, this is the first time that an end-to-end method can outperform those using the stage optimization strategy.

In terms of modality usage, our method just uses extra pose modality during training, while only RGB videos are needed for inference. Thus, it is simpler for real application compared to DNF (Cui et al., 2019) which is built on a two-stream architecture taking both RGB videos and optical flow as inputs.

Finally, the results reported on CSL-Daily may be more important due to its large vocabulary size. Our method can still achieves SOTA performance on this large-scale dataset, which also validates the generalization capability of our method over different sign languages.

As shown in Table 10(b), we further evaluate our SRM on the following two signer-independent benchmarks: PHOENIX-2014-SI and CSL.

Although some works, e.g., DNF (Cui et al., 2019) and CMA (Pu et al., 2020), evaluate their method on PHOENIX-2014-SI, none of them propose any dedicated module to deal with the challenging SI setting. In this work, we develop a simple yet effective signer removal module (SRM) for SI-CSLR to make the model more robust to signer discrepancy. As shown in Table 10(a), our $\text{C}^{2}$SLR can already achieve competitive performance on PHOENIX-2014-SI, and the SRM can further improve the performance significantly. The result validates that feature disentanglement is an effective method to remove signer-relevant information, and we believe our SRM can serve as a strong baseline for future works on SI-CSLR.

As shown in Table 10(b), our SRM can lead to a relative performance gain of 24.4% over the baseline $\text{C}^{2}$SLR on CSL⁵⁵5Although the SI setting itself is challenging, since the sentences in the CSL test set all appear in the training stage, the WER can be very low ($<1$%).. It is worth noting that the SOTA work, MSeqGraph (Tang et al., 2021), uses three modalities including RGB, pose, and depth. However, our method only uses RGB and pose information for training, and only RGB frames are needed for inference. Thus, with a comparable performance to the SOTA work, we believe our method is more applicable in real practice.