Hand1000: Generating Realistic Hands from Text with Only 1,000 Images

Text-based image generation has long been a prominent research area, with classical methods including Generative Adversarial Networks (GANs) and their variants (Kang et al. 2023; Tao et al. 2022; Xu et al. 2018; Zhu and Ngo 2020; Ye et al. 2021; Zhu et al. 2019b, a; Wang et al. 2024; Li et al. 2023a; Ma et al. 2023), as well as stable diffusion models and their derivatives (Sauer et al. 2023; Saharia et al. 2022b; Dhariwal and Nichol 2021; Ho, Jain, and Abbeel 2020; Ho et al. 2022; Sohl-Dickstein et al. 2015; Yang et al. 2024; Ye et al. 2024; Yu et al. 2024). These methods leverage the capabilities of CLIP (Radford et al. 2021) and Transformer (Vaswani et al. 2017), empowering them to generate images from textual descriptions. In recent years, numerous related studies have emerged, achieving notable success in the field such as GigaGAN (Kang et al. 2023), Imagen (Saharia et al. 2022b), and SDXL Turbo (Sauer et al. 2023). With the advent of ChatGPT (Brown et al. 2020; Achiam et al. 2023) and the development of large language models(LLM) (Chowdhery et al. 2023; Touvron et al. 2023b; Team et al. 2023), several multimodal models for text-to-image generation have also emerged, such as DALL-E (Ramesh et al. 2021). Despite the notable advancements these methods have achieved in image generation, they frequently encounter significant issues when producing images that include human hands, often resulting in severe distortions or anomalies in hand representations.

Text-based diffusion models are effective for generating various images but struggle with rendering human hands. Recent works address this challenge in two ways. Some studies use image inpainting, like HandRefiner (Lu et al. 2023), which identifies and crops hand regions, then refines them using ControlNet (Zhang, Rao, and Agrawala 2023). However, this approach depends on another model for initial image generation and fails with severely distorted hands. Alternatively, methods like HanDiffuser (Narasimhaswamy et al. 2024) directly generate hand images by reconstructing text prompts into 3D hand parameters, though the training process is complex and expensive. Additionally, the above methods require an enormous training dataset, in the order of hundreds of thousands of samples, to achieve noticeable results. In contrast, our Hand1000 method fine-tunes on just 1,000 images to generate realistic hands without requiring explicit hand pose modeling, offering a more efficient and flexible approach.

Text-based image editing has seen rapid advancements in recent years, allowing for the modification of an image’s background, style, subject size, and movements through direct textual input. These technologies, often derived from improvements in GANs (Abdal et al. 2021; Härkönen et al. 2020; Lang et al. 2021; Patashnik et al. 2021; Shen et al. 2020; Shen and Zhou 2021; Xia et al. 2023) or Stable Diffusion models (Ruiz et al. 2023; Kawar et al. 2023; Nichol and Dhariwal 2021; Rombach et al. 2022; Saharia et al. 2022a, c; Song, Meng, and Ermon 2020; Song and Ermon 2019, 2020), typically require minimal input images to achieve the desired effects. For instance, Imagic (Kawar et al. 2023) firstly fine-tunes text embeddings, then fine-tunes the diffusion model, and finally performs interpolation. This method focuses on text embeddings, and achieves remarkable results. Considering the challenge of generating realistic hand images, it seems plausible to borrow from the aforementioned image editing concepts. However, image editing algorithms cannot be directly applied to generate realistic hand images because they fundamentally rely on the diffusion model’s inherent capabilities, which lack prior knowledge of realistic hands. Besides, these methods usually take several minutes to modify a single picture, making it impossible to generate a large amount of images in a short time. To address these problems, we propose Hand1000, a method that fine-tunes the model using image editing principles. This approach not only imparts the model with knowledge of realistic hands but also benefits from the characteristics of image editing algorithms, requiring fewer training images and faster training speed.

Text-to-image generation aims to generate an image that accurately represents a given textual description. The process of a standard text-to-image diffusion model (Rombach et al. 2022) is as follows: Given an initial noise image $\epsilon\sim\mathcal{N}(0,I)$, where the noise follows a Gaussian distribution, the CLIP text encoder (Radford et al. 2021) $\Gamma$ encodes the text into a text embedding $\textit{e}=\Gamma(\textit{text})$. Using e and the time step t as conditions, the diffusion model generates a latent representation $\mathbf{z}_{t}:=\alpha_{t}\mathbf{x}+\sigma_{t}\epsilon$ based on the initial noise image $\epsilon$. Subsequently, an image is generated from this latent representation. The entire process is guided by the following mean squared error loss:

$$\mathbb{E}_{\mathbf{x},\mathbf{e},\epsilon,t}\left[w_{t}\left\|\hat{\mathbf{x}}_{\theta}(\alpha_{t}\mathbf{x}+\sigma_{t}\epsilon,\mathbf{e})-\mathbf{x}\right\|_{2}^{2}\right],$$

(1)

where $\mathbf{x}$ represents the real image, and $\mathbf{e}$ denotes the text embedding. The terms $\alpha_{t}$, $\sigma_{t}$, and $w_{t}$ are parameters that control the noise scheduling and sample quality, and they are functions of the diffusion process time $t\sim U([0,1])$.

Assume we have a set of images depicting a target hand gesture, such as “phone call”. To extract hand features associated with this gesture, we feed the images into a gesture recognition model(i.e. Mediapipe hands (Zhang et al. 2020)) to obtain features from the final layer of the network. Subsequently, these features are averaged to obtain a Mean Hand Gesture Feature representation of the gesture, which is used for training in the following stages. Simultaneously, the Mean Hand Gesture Feature is also preserved and utilized during the inference phase.

In the second stage of training, the main objective is to integrate text embedding with hand gesture features to facilitate the diffusion model to generate realistic hand images. Given a textual description as input, CLIP text encoder is employed to obtain the Text Embedding. Subsequently, the text embedding is concatenated with the Mean Hand Gesture Feature obtained from the first stage. To fuse the text and hand gesture features, a fully connected (FC) layer is employed to map the concatenated embedding to the same dimension of text embedding, resulting in the Fused Embedding, which smoothly incorporates the gesture’s general characteristics. Notably, the FC layer is kept frozen to reduce computational costs, as the embedding will undergo further optimization in Stage II. A linear fusion of the fused embedding with the original text embeddings is subsequently performed to obtain the Double Fused Embedding:

$$e_{d}=\lambda\times e_{t}+\left(1-\lambda\right)\times e_{f},$$

(2)

where $\mathbf{\lambda}$ is the hyperparameter, $\mathbf{e_{d}}$ is Double Fused Embedding, $\mathbf{e_{t}}$ is Text Embedding, and $\mathbf{e_{f}}$ is Fused Embedding. This linear fusion reinforces the text embedding’s unique features, allowing it to leverage the gesture feature without overpowering its own nuances.

Inspired by image editing techniques (Kawar et al. 2023), we proceed to optimize the Double Fused Embedding to better align with the real hand images. Specifically, the stable diffusion model is kept frozen while the Double Fused Embedding is fine-tuned using the loss function defined in Equation 1, resulting in what we term Optimized Embedding. The Optimized Embedding, enriched and refined with hand gesture information, establishes a stronger alignment with the corresponding hand images, making it ideal for the subsequent hand image generation in Stage III.

In the final stage of training, the Optimized Embedding obtained in Stage II is used as a condition to fine-tune the Stable Diffusion model with the loss function specified in Equation 1. The fine-tuning procedure follows the standard text-to-image diffusion model training, with the only modification being the replacement of the usual text embedding with our Optimized Embedding from Stage II. To preserve the hand gesture information integrated in Stage II, the Optimized Embedding is kept frozen during this stage, in line with image editing works (Kawar et al. 2023).

As illustrated in Figure 3, during the inference phase, a textual description containing a hand gesture is used as input. Similar to the training stage, the CLIP text encoder is adopted to obtain the Text Embedding. The Mean Hand Gesture Feature, stored from during the Stage I in the training phase, is concatenated with the Text Embedding. This concatenated feature is then reduced in dimension using a fully connected layer to obtain the Fused Embedding. The Text Embedding and the Fused Embedding are linearly fused to produce the Double Fused Embedding:

$$e_{d}=\mu\times e_{t}+\left(1-\mu\right)\times e_{f},$$

(3)

where $\mathbf{\mu}$ is the hyperparameter, $\mathbf{e_{d}}$ is Double Fused Embedding, $\mathbf{e_{t}}$ is Text Embedding, and $\mathbf{e_{f}}$ is Fused Embedding.The Double Fused Embedding is then used as a condition to input into the trained Stable Diffusion model in Stage III, enabling the generation of images that incorporate the specified gesture and maintain a realistic hand appearance.

As no existing datasets specifically designed for text-to-hand image generation are publicly available, we construct a new dataset to bridge this gap. The process of the dataset curation is illustrated in Figure 4. Our dataset is built upon HaGRID (Kapitanov et al. 2024)), which is a comprehensive collection of various hand gestures. This dataset features clear hand images and includes a diversity of individuals, clothing, and colors, making it highly suitable for our training and evaluation purposes. We construct our datasets using six hand gestures—“phone call,” “four,” “like,” “mute,” “ok,” and “palm” from HaGRID. 1,000 images are selected for each gesture for training and testing respectively. These hand gestures include ones with high finger separation (e.g., “palm”), tightly closed fingers (e.g., “mute”), and complex poses (e.g., “ok”), providing a comprehensive assessment of the model’s capabilities.

Nevertheless, HaGRID (Kapitanov et al. 2024)) only contains hand gesture labels and lacks detailed textual descriptions of the images, which can not be directly used to train our model. To address this issue, we first adopt image captioning models to generate a textual description for each image. To inject hand gesture information into the textual description, we input the gesture labels and textual descriptions into the LLaMA3 model (Touvron et al. 2023a), enriching the original text with rich hand gesture information. We employ three different image captioning models—BLIP2 (Li et al. 2023b), PaliGemma (Beyer et al. 2024), and VitGpt2 (Mishra et al. 2024)—to generate the textual descriptions. To evaluate the quality of the textual descriptions, we use CLIP’s (Radford et al. 2021) text and image encoders to calculate the similarity between the textual descriptions and corresponding images as well as the pairwise similarity between texts. As shown in Table 1, the results show that the text-image similarity improved after postprocessing with LLaMA3 (Touvron et al. 2023a), while the textual similarity does not increase significantly, demonstrating the effectiveness of this approach. According to Table 1, BLIP2 (Li et al. 2023b) shows the best performance, which is thus chosen as the image captioning model for generating textual descriptions for our dataset.

we employ five evaluate metrics to assess the quality of the generated images: Frechet Inception Distance (FID), Kernel Inception Distance (KID), FID-H, KID-H, and HAND-CONF (Heusel et al. 2017; Parmar, Zhang, and Zhu 2022). FID-H involves using an open-source hand gesture recognition model (i.e. Mediapipe hands (Zhang et al. 2020)) to segment the hand regions in the images, and then computing the FID on these segmented patches; KID-H is calculated similarly. The first four metrics measure the distance between the feature distributions of generated images and real images, with smaller distances indicating that the generated images are closer to real ones and thus of higher quality. HAND-CONF is computed by running a pre-trained hand detector(i.e. Mediapipe hands (Zhang et al. 2020)) and calculating the confidence scores of the detections. Higher confidence scores suggest that the detector is more certain about a region being a hand, indicating better quality of hand generation.

For each image in the training set, the first stage of training involves 10 epochs with a learning rate of 1e-3. The second stage consists of 20 epochs with a learning rate of 1e-6. Besides, We use Mediapipe hands (Zhang et al. 2020) as the hand gesture recognition model while the text encoder is the CLIP (Radford et al. 2021) text encoder. The sampling scheme is DDIM (Song, Meng, and Ermon 2020) and the optimizer is Adam (Kingma 2014). We used SDv1.4 for our experiments, which is similar to recent works like HanDiffuser (Narasimhaswamy et al. 2024), HandRefiner (Lu et al. 2023), and Imagic (Kawar et al. 2023).

As shown in Table 2, our Hand1000 model demonstrates a significant performance improvement over the Stable Diffusion (Rombach et al. 2022) in terms of all the metrics. Specifically, Hand1000 achieves a 28.599 reduction in FID, a 0.017 reduction in KID, a 30.208 reduction in FID-H, and a 0.041 reduction in KID-H, while its HAND-CONF score is elevated by 0.053. To further validate the effectiveness of our method, we also perform comparison with fine-tuned stable diffusion using the same dataset. Although the fine-tuned Stable Diffusion model demonstrates slightly superior performance in terms of FID and KID compared to our method, several studies (Kynkäänniemi et al. 2019; Borji 2019) have suggested that these two metrics are not entirely reliable. Nevertheless, the quality of hand image generation is significantly inferior, as indicated by FID-H, KID-H, and HAND-CONF. Compared to the fine-tuned stable diffusion model, our method achieves a 44.294 reduction in FID-H, a 0.027 reduction in KID-H, and a 0.035 increase in HAND-CONF, underscoring the effectiveness of our approach. Furthermore, we also conduct experiments on the open-source hand image refinement method, HandRefiner (Lu et al. 2023), and the results demonstrate that our method outperforms HandRefiner (Lu et al. 2023) across all metrics, with an 11.396 reduction in FID, a 0.006 reduction in KID, a 10.037 reduction in FID-H, a 0.003 reduction in KID-H, and a 0.029 increase in HAND-CONF.

By providing identical text prompts, we conduct a comparative analysis of the results with Stable Diffusion, finetuned Stable Diffusion, using Imagic (Kawar et al. 2023) to edit the results of Stable Diffusion and fine-tuned Stable Diffusion as well as HandRefiner. As illustrated in Figure 5, our method not only generates realistic hand images but also effectively renders other components such as characters and clothing, demonstrating superior performance compared to all the other methods. Furthermore, though Imagic is adopted to edit the image generated by Stable Diffusion, the result led to subtle alterations, such as the lightening of the dress color in the first set of images in the bottom row. In other words, Imagic fails to recover realistic hands and bodies from the generated images, leaving them in a chaotic, deformed, and distorted state. Additional experimental results can be found in the supplementary materials. Besides, HandRefiner fails to interpret gesture information accurately, despite some improvement in hand deformation, and shows no effect when only partial hands are present, as in the rightmost example.

To validate the effectiveness of our design choices, we present a series of ablation experiments, including the ablation of various main components of the method and the ablation of several hyperparameters.