DiffSign: AI-Assisted Generation of Customizable
Sign Language Videos With Enhanced Realism

Our approach for generating sign language videos combines a parametric modeling approach to generate high-fidelity poses with a generative modeling approach to synthesize customizable, realistic-looking signers. Fig. 1 provides an overview of our approach. We first extract the 2D pose sequence from a human sign language video for a media content and retarget the sign pose sequence onto a 3D avatar model by optimizing a parametric model. Retargeting to 3D avatars helps in capturing and transferring high-fidelity signing poses even when the intermediate video frames in the source video are blurry. We then leverage recent advances in diffusion-based generative models to generate user customizable sign language videos with enhanced realism without additional training, by conditioning on the poses extracted from the rendered 3D avatars.

We experimented with video-based diffusion models (e.g. ControlVideo[20]) to directly generate sign language videos by conditioning on the sign poses. However, these methods were able to generate only a few seconds of video. Moreover, the pose transfer was inaccurate and had very low fidelity. Hence, the resulting sign language video was not interpretable. Instead, our approach generates the sign language video frame-by-frame by conditioning on the pose and appearance inputs.

One of the challenges in performing this conditional generation on a per-frame level using text-to-image diffusion models, such as Stable Diffusion[11], is that it results in temporal inconsistency, since the appearance of the signer changes in each frame despite using the same seed and text prompt, as shown in Fig. 2, where the shirt and beard of the signer varies in the frames. We address this challenge by conditioning on a single image of the signer, which is input to a diffusion model using a visual adapter, when generating each frame. Our results in Section 5 show that this generation approach using a visual adapter results in generating signing videos with realistic-looking signers that have more consistent appearance across the frames, regardless of the duration of the media content. The signer depicted in the visual prompt can be further customized using text and other multimodal prompts to generate personalized signing videos.

We start with the pose extraction step, as shown in Step 1 in Fig. 3a. Given a human sign language video, we used the MediaPipe library[7] to extract the 2D poses of the signer that are relevant to sign language generation from each video frame. We also experimented with OpenPose[3], but found it to be significantly slower. The poses relevant to sign language generation include facial, hand, finger, and upper body poses. We extracted facial keypoints, 25 upper body keypoints, and 21 keypoints per hand, including the fingers. MediaPipe was unable to extract some of the keypoints from blurry frames where there was rapid hand motion. For such frames we linearly interpolated the 2D keypoints from neighboring frames. Although we directly estimate poses from the frames of a human sign language video currently, we can also use the sign pose sequence that is output by a trained language2pose model, as mentioned earlier.

The frame-by-frame pose extraction causes considerable jitter in the resulting video due to keypoint shift. To smooth the poses and reduce jitter, we used the 1-€ (one-euro) algorithm [4], which is a lightweight, low-pass noise filtering algorithm that is based on adaptive exponential smoothing. This algorithm uses two hyper-parameters: the speed coefficient, $\beta$, to control the lag and minimum cutoff frequency, $f_{min}$, to control the jitter. Based on our experiments, $\beta=1.0$ and $f_{min}=0.04$ resulted in a good balance between jitter and lag for our data.

Next, we retargeted the filtered 2D poses onto a 3D human avatar, represented by the SMPL-X parametric human body model[9]. SMPL-X is defined by a function $M(\theta,\alpha,\phi)$, where $\theta$, $\alpha$, and $\phi$ represent the pose, shape, and facial expression parameters, respectively. We chose SMPL-X because the pose and shape parameters model the body, hands, and fingers, which are crucial for sign language generation. Pose retargeting involves fitting the SMPL-X body model to the 2D keypoints extracted from RGB images by optimizing the parameters. We used the SMPLify-X[9] algorithm for retargeting the sign poses extracted from each frame in the previous step to the SMPL-X avatar model.

We had to address a couple of challenges when using SMPLify-X. First, SMPLify-X requires full body keypoints for optimization, while the sign language videos only depict upper body, as shown in Fig. 1. Optimizing only the upper body poses resulted in invalid deformations, even though SMPLify-X uses a body pose prior for minimizing the deformation. However, the use of MediaPipe allowed us to extract the full body keypoints with some degree of confidence from upper body videos and we optimized only the pose and expression parameters, while freezing the shape parameters, to avoid body deformations. We also tried with OpenPose, but were not able to extract full body keypoints from upper body signing videos using OpenPose. Second, since SMPLify-X is trained on OpenPose keypoints, whereas we used MediaPipe to extract the keypoints, we had to first map the MediaPipe keypoints to the appropriate OpenPose keypoints. For example, MediaPipe extracts more than 400 detailed facial keypoints, whereas OpenPose extracts only around 70 facial keypoints. So we mapped only a subset of the MediaPipe facial keypoints to the corresponding OpenPose keypoints and optimized them with SMPLify-X.

Pose retargeting results in a 3D signing avatar corresponding to each input frame, as depicted in Step 3 in Fig. 3a. In order to further reduce jitter in the resulting video, we filtered the vertices of the 3D avatar mesh generated for each frame using the 1-€ filtering approach. Finally, each avatar mesh was rendered with appropriate texture map and lighting using Blender[1] to generate the sign language poses as a sequence of RGB frames, as shown in Step 5 in Fig. 3a.

The last step in the pipeline is to generate a synthetic human signer and transfer the sign poses from the avatar generated in the previous step to the synthetic human signer. The appearance and pose of the synthetic signer are decoupled and controlled separately, as shown in Fig. 3b.

One way to control the appearance of the signer is by leveraging a text-to-image diffusion model and describe the appearance of the signer using only a text prompt. However, this resulted in inconsistent signer appearance across the generated frames despite using the same seed for each frame, while the poses were transferred with high fidelity from the avatar frames, as shown in Fig. 2. Instead, we leveraged a lightweight visual adapter (IP-adapter)[18] and used a single image of the target signer as an input prompt. The visual adapter extracts features from the image prompt using a CLIP encoder[10], which are then used to condition the generation of the synthetic signer using a pre-trained Stable Diffusion model[11]. While the pre-trained model has a cross-attention module to provide context from a text prompt, the visual adapter adds a dedicated cross-attention module to input context based on the image features to the U-net layers of the pre-trained diffusion model, in order to compute the attention scores for generating the synthetic signer. This image-based generation approach significantly improved the temporal consistency of the signer appearance across the frames. To further customize the signer appearance, a text prompt or other multimodal prompts can be optionally used in conjunction with the image prompt, as shown by the results in Section 5.

While the appearance is conditioned on the image prompt, in order to generate the signer with the appropriate sign pose, we extracted canny edges [2] and poses for the face, hands, fingers, and upper body from each avatar frame rendered in the previous step and used them as task-specific inputs to ControlNet, as shown in Fig. 3b. These conditional inputs are used by Stable Diffusion for high-fidelity, frame-by-frame zero-shot pose transfer to the generated signer. The frames are then concatenated to generate the sign language video.

Fig. 4 compares a subset of the frames from the videos generated using different approaches with the corresponding frames in the ground-truth sign language video (shown in Row 1), which was created by a real human signer for a stand-up comedy media content. Row 2 shows the video frames from the avatar video which was rendered after retargeting the human signer poses using the parametric approach presented in Section 4.

Rows 3 and 4 in Fig. 4 show the sign language videos with realistic-looking synthetic signers that were generated using a diffusion-based approach, while conditioning on the avatar poses supplied through ControlNet. The video frames in Row 3 were generated using the same seed by conditioning on the text prompt ("a young male with beard wearing a white shirt"), that was input to a pre-trained text-to-image Stable Diffusion XL model, in addition to conditioning on the avatar poses from Row 2. Even though the frame-by-frame pose transfer is quite accurate, the appearance of the synthetic signer varies across the frames, especially for long videos, resulting in a jittery video. For example, the frames in Fig. 2 and in Row 3 of Fig. 4 are from different time segments of the same video consisting of 1380 frames and the inconsistent appearance is clearly evident.

Instead of conditioning on a text prompt, the frames in Row 4 were generated using our approach that uses a visual adapter to input an image prompt to a pre-trained Stable Diffusion model in order to control the signer appearance. The signer poses were conditioned on the avatar sign poses (shown in Row 2) that were input through ControlNet. Using the same visual conditioning for each frame significantly improves the temporal consistency of the signer across the video frames, resulting in lower jitter, even without explicit smoothing, instance-specific training on the target signer images, or training on sign language content.

In the previous section, we compared two diffusion-based approaches for generating synthetic sign language videos, one of which used a text prompt while the other used a visual conditioning approach to control the appearance of the synthetic signer generated by a pre-trained Stable Diffusion model. In both cases, the diffusion-based generative model was used in inference mode for zero-shot pose transfer. It was not trained or fine-tuned on the target signer images.

In this section, we present results of a third approach for personalizing the appearance of the synthetic signer, which involves fine-tuning the pretrained Stable Diffusion model with just a few images of the target signer. Our goal was to determine if fine-tuning improves the generation quality. To do this, we used DreamBooth[12], which is a method for fine-tuning pre-trained, large-scale, text-to-image diffusion models with just a few images of a personalized subject, by associating a unique identifier with that subject. The fine-tuned model can then be used to generate the subject in personalized contexts by embedding the unique identifier in the text prompt provided as a conditional input for the generation. We fine-tuned StableDiffusion v1.5 using DreamBooth with 5 frames from the synthetic signer video shown in Row 4 of Fig. 4 and associated these images with a random unique identifier, "<sks> Edward" to personalize the signer. Then we used this fine-tuned model to generate the sign language video by conditioning the signer appearance on a simple personalized text prompt which includes the identifier associated during fine-tuning: "<sks> Edward wearing a black shirt". Similar to the results in Fig. 4, zero-shot pose transfer to this personalized signer was achieved by supplying the canny edges and poses extracted from the avatar video as task-specific inputs through a pre-trained ControlNet model.

Fig. 5 shows the results of the fine-tuned model. We see that the model that was fine-tuned on just 5 images of the target signer and conditioned on a simple personalized text prompt is able to generate a signer whose appearance closely matches the target signer (shown in Row 4 of Fig. 4). However, the appearance of the signer in Fig. 5 is not consistent across frames, despite personalization. For example, there is no beard in the first frame although all the 5 images used for fine-tuning had a beard. In comparison, our approach of supplying a single image of the target signer through a visual adapter to control the signer appearance results in a sign language video with better temporal consistency even without fine-tuning, as shown in Row 4 of Fig. 4.

Fig. 4 presented a qualitative comparison of the sign language videos with synthetic signers generated using parametric and diffusion-based generative approaches. In this section, we use different metrics to quantitatively evaluate the temporal consistency, alignment with prompt semantics, and the realism of the synthetic sign language videos generated using the different models.

1) Structural Similarity (SSIM): We computed the structural similarity (SSIM) between consecutive pairs of frames to measure the jitter in the generated video. In Table 1 we have reported the average SSIM computed over the entire video having 1380 frames. Higher the SSIM value, lower is the jitter and hence, better is the consistency or smoothness of the generated video. We see that the video output by our generative approach based on visual conditioning (frames shown in Row 4 of Fig. 4) has an SSIM value of 0.769. Fine-tuning the diffusion model with a few images of the target signer and conditioning the generation on a personalized text prompt, as described in Section 5.2, results in SSIM=0.668, while conditioning a pre-trained diffusion model on a text prompt without any fine-tuning results in SSIM=0.553. These results show that our approach of visually conditioning the signer generation on just a single signer image using a pre-trained model with a visual adapter, results in lower jitter compared to the text conditioned models and outperforms even the fine-tuned model. Furthermore, by comparing the two text-conditioned approaches we infer that fine-tuning on a few target signer images helps lower the jitter.

Finally, we see that even without any pose smoothing or frame interpolation, the video generated using our visual conditioning approach (with SSIM=0.769) has only slightly higher jitter than the avatar video (with SSIM=0.812) that is rendered after explicit pose smoothing using the 1-€ algorithm. To summarize, conditioning the signer generation on just a single image of the target signer supplied through a visual adapter improves the temporal consistency of the sign language video, without the need for fine-tuning on additional signer images, complex prompt engineering, or explicit pose smoothing. The consistency of the generated video may be further improved using frame interpolation methods.

2) Directional Similarity (DS): The transfer of sign poses from a human signer video to a synthetic signer or avatar video effectively results in a change in the signer appearance while preserving the poses. Hence, this can be regarded as a type of style transfer or editing operation where some aspects of the original content are modified and one way to quantitatively evaluate the generated sign video is by measuring how well it conforms to the editing instructions. Given a pair of images $(i_{s},i_{t})$ and text descriptions $(d_{s},d_{t})$, where $d_{s}$ describes the source image $i_{s}$ and $d_{t}$ describes the target image $i_{t}$ resulting from editing $i_{s}$, the directional similarity measures how well the change in the edited image aligns with the editing semantics. To compute this, we encoded the images using the CLIP image encoder[10], $E_{i}$, and the text descriptions using the CLIP text encoder, $E_{d}$. We then computed DS using cosine similarity between the encoded image and text, as follows: $DS=cosineSim((E_{i}(i_{t})-E_{i}(i_{s})),(E_{d}(d_{t})-E_{d}(d_{s})))$.

We computed the frame-wise DS between the human sign language video, that is regarded as the source (Row 1 in Fig. 4) and the synthetically generated signer video, regarded as the target. The average DS over the entire video for each of the generative methods is reported in Table 1. To compute the DS, a simple high-level description was used for all the frames of the real human source video: $d_{s}$ = "a female sign language signer". Similarly we used a high-level description for all the frames of the generated video to match the appearance of the synthetic signer. For e.g. the video frames generated by the visual prompt based diffusion method (shown in Row 4 in Fig. 4) were described by $d_{t}$ = "a young male sign language signer with a beard wearing a white shirt". From Table 1 we see that the visual adapter based method has the highest directional similarity. Even though this method was conditioned only on a single image prompt and did not use any text prompts like the other two diffusion-based methods in the table, it is able to generate a signer whose appearance is more consistently aligned with the semantics of the style editing descriptions that indicate that the sign poses are transferred from a female signer (source) to a bearded, male signer (target).

3) Frechet Inception Distance (FID): To compare the realism and visual quality of the generated videos, we computed the FID score[6] with respect to the human sign language video (Row 1 of Fig. 4) using the FID implementation in the torchmetrics library. A lower FID score signifies higher visual quality and realism of the signer appearance. Based on the FID scores in Table 1, the avatar video has the highest FID score=176.035, as expected, since the avatar has the least human-like appearance (see frames in Row 2 of Fig. 4). In comparison, the video generated by our visual conditioning approach has a lower FID score=146.49, indicating better visual quality and realism compared to the avatar video and the video generated using the pre-trained diffusion model conditioned on a text prompt (FID=158.525). Fine-tuning the diffusion model with a few images of the target signer helps in further improving the visual quality of the signer (see Fig. 5) and results in the lowest FID value=130.896. Evaluation of the synthetic signing videos using human subjects will be part of future work.

Fig. 6, shows the impact of using alternative task-specific inputs to ControlNet for conditioning the signer generation. From Fig. 6a, we see that if we use only the poses extracted from the avatar video as the conditional inputs, the resulting poses are incorrect and in some cases, do not even represent sign language poses. On the other hand, Fig. 6b shows that conditioning on a combination of canny edge and depth map transfers the poses correctly, but results in inconsistent signer appearance across different frames. We found that conditioning on a combination of canny edges and pose input results in more accurate pose transfer to the generated signer, as shown in Fig. 4, with the contours of the body parts provided by the canny edge detection contributing to the improvement.

In order to achieve our goal of allowing users to customize the appearance of the signer based on their preferences, our approach supports multimodal prompts.

As stated earlier, we input the canny edge and pose inputs through ControlNet to condition the signer poses and an image prompt through the visual adapter to control the base appearance of the synthetic signer. Additionally, a text prompt can be supplied to Stable Diffusion to further customize the appearance of the target signer. Fig. 7 shows how a text prompt in conjunction with an image prompt is used to generate sign language videos with the same signer wearing glasses in Row 2 and wearing a blue-colored shirt in Row 3, instead of a white-colored shirt.

Fig. 8 shows how we generated sign language videos for the same media content using diverse synthetic signers, in order to make the content more accessible to different communities. Each row shows a subset of frames from the sign language video for the same media content using a different synthetic, but realistic-looking signer.

First, we used simple text prompts to generate a base image of synthetic human signers that are diverse in age group, demography, gender, and race using a Stable Diffusion model. Fig. 8 shows two examples of these synthetically generated target signers in the leftmost column, with a young boy in Row 2 and an African-American woman in Row 3. Then this base image is input as a visual prompt through the visual adapter, in order to control the appearance of the signer in the generated signing video. The avatar poses, as shown in the topmost row, are used as conditioning inputs for frame-by-frame zero-shot transfer of the poses to the synthetic signers. These diverse synthetic signers can be used to sign the video content targeting different DHH communities. For example, the young boy as a synthetic signer can be used for signing kids shows and cartoons.