Semi-supervised Chinese Poem-to-Painting Generation via Cycle-consistent Adversarial Networks

Image-to-text translation aims to generate natural language descriptions from visual input. Early approaches relied on template-based methods that filled in handcrafted templates with detected visual concepts [4, 5, 6]. More recently, deep learning models have achieved significant progress by learning the mapping between images and text in an end-to-end manner [7, 8, 9].

Encoder-decoder architectures have emerged as a popular choice for image-to-text translation, where convolutional neural networks (CNNs) are employed to encode the image into dense feature vectors, and recurrent neural networks (RNNs) are used to decode these features into word sequences [7]. The incorporation of attention mechanisms has further enhanced the performance of these models by enabling them to selectively focus on relevant image regions when generating each word [8]. Hierarchical approaches have also been proposed to decompose the generation process into multiple stages, such as first predicting a semantic layout and then filling in the details [10, 11, 12]. Additionally, adversarial training techniques have been explored to improve the naturalness and diversity of the generated captions [13].

Beyond generating purely descriptive captions, some works have explored more artistic and stylized text generation from images, such as composing poetry [14, 15] or generating stylized captions [16, 17]. However, these approaches often rely on paired image-text datasets for fully-supervised training, which limits their applicability to niche domains like classical Chinese art where such paired data is scarce. To address this limitation, unsupervised or weakly-supervised approaches that can learn from unpaired image and text data have gained attention in recent years [18]. These methods typically employ techniques such as cycle consistency [3] or adversarial alignment [19] to bridge the gap between the visual and textual domains, enabling the generation of stylized or artistic descriptions even in the absence of paired training data.

Text-to-image generation aims to synthesize images from natural language descriptions. Generative adversarial networks (GANs) [20, 21, 22] revolutionized this field, enabling the generation of realistic images conditioned on textual input. GANs formulate the problem as a minimax game between a generator network that aims to synthesize realistic images from input texts, and a discriminator network that tries to distinguish between real and generated images. The generator is trained to fool the discriminator, while the discriminator is trained to improve its classification accuracy. This adversarial training paradigm has led to significant improvements in the quality and diversity of generated images.

Reed et al. [23] first proposed an end-to-end GANs architecture for text-to-image synthesis, which learned a direct mapping from textual descriptions to images. However, the generated images often lacked fine-grained details and consistency with the input texts. Subsequent works have focused on improving the visual quality and semantic alignment of the generated images through various techniques, such as attention mechanisms [24], multi-stage refinement [25], and hierarchical generation [26]. Qiao et al. [27] proposed a mirror structure that reconstructed the input text from the generated image, encouraging semantic consistency. To overcome the paired data limitation, some recent works have explored unsupervised or semi-supervised approaches. Gu et al. [28] proposed a cycle-consistent adversarial network that learns bidirectional mappings between the text and image domains, enabling text-to-image synthesis without paired data. Huang et al. [29] introduced a semi-supervised approach that leverages both paired and unpaired data using a hierarchical alignment strategy.

Beyond GANs, several other advanced approaches have recently emerged for text-to-image synthesis. Diffusion models, such as Denoising Diffusion Probabilistic Models (DDPM) [30], have shown impressive results in generating high-quality images from text. DDPM learns to iteratively denoise a Gaussian noise input conditioned on the text embedding, generating realistic images through a gradual refinement process. Autoregressive models, like DALL-E [31], have also demonstrated remarkable performance in text-to-image generation. DALL-E uses a transformer architecture to autoregressively predict image tokens conditioned on the input text, enabling the generation of diverse and semantically consistent images. CogView [32] is another powerful text-to-image model that combines the strengths of autoregressive transformers and variational autoencoders. It learns a joint distribution over text and image tokens, allowing for controllable and high-quality image generation guided by textual descriptions.

For artistic text-to-image synthesis, such as painting generation from poetry, preserving the artistic style and abstract content is essential. Zhu et al. [33] introduced a memory-based model that selectively uses learned artistic strokes and textures to compose paintings that match the poetic descriptions. Xue et al. [34] proposed Sketch-And-Paint GAN (SAPGAN), the first end-to-end model for generating Chinese landscape paintings without conditional input, using a two-stage GAN architecture to generate edge maps and translate them into paintings. Fu et al. [35] introduced a Flower-Generative Adversarial Network framework to generate multi-style Chinese flower paintings, using attention-guided generators and discriminators, and a novel Multi-Scale Structural Similarity loss to preserve image structure and reduce artifacts.

To alleviate the shortage of pairwise training data, we adopt cycle-consistent adversarial networks to enable semi-supervised learning as shown in Figure 5. The framework consists of four sub-networks: the painting encoder $E_{p}$, the poem generator $G_{t}$, the poem encoder $E_{t}$, and the painting generator $G_{p}$. The encoders map inputs into a shared latent space, while the generators decode latents into the corresponding output domain.

As shown in Figure 4, the encoders and generators are trained to minimize the cycle-consistency loss, which measures the reconstruction error between the original inputs and their cycle-reconstructions:

where $x$ and $y$ denote samples from the painting and poem domains, respectively. For unpaired samples, only the cycle-consistency loss is applied as the ground-truth targets are unknown.

When paired data $(x_{i},y_{i})$ is available, we introduce additional supervised losses to penalize deviations from the ground-truth:

To encourage generated outputs to match the distributions of real data in each domain, we employ adversarial losses [20]. For poem generation, we introduce a sequence discriminator $D_{t}$ that aims to distinguish real poems from generated ones. The adversarial loss for the poem generator $G_{t}$ is defined as:

Similarly, a painting discriminator $D_{p}$ is used to assess the realism of generated paintings, with the adversarial loss for the painting generator $G_{p}$ defined as:

The full objective for the cycle-consistent adversarial framework is:

where $\lambda_{sup}$ and $\lambda_{adv}$ are weights that control the relative importance of the supervised and adversarial losses.

This cycle-consistent adversarial framework allows the model to learn bidirectional mappings between the painting and poem domains by leveraging both paired and unpaired data. The adversarial training ensures that generated outputs are plausible and indistinguishable from real data.

The image-to-text translation module generates diverse poems from input Chinese paintings, as shown in Figure 4. It comprises an image encoder $E_{p}$, a poem generator $G_{t}$, and a sequence discriminator $D_{t}$.

The image encoder $E_{p}$ is a CNN that extracts high-level visual features $\mathbf{v}\in\mathbb{R}^{d_{v}}$ from an input painting $x$. The poem generator $G_{t}$ is a LSTM-based network[38], that synthesizes a poem $\hat{y}=(\hat{y}_{1},\ldots,\hat{y}_{T})$ of length $T$ conditioned on the visual features:

where $h_{t}\in\mathbb{R}^{d_{h}}$ is the hidden state at time step $t$, $\mathbf{e}(\cdot)$ is a word embedding function that maps each token to a dense vector, $\mathbf{W}_{o}\in\mathbb{R}^{|V|\times d_{h}}$ and $\mathbf{b}_{o}\in\mathbb{R}^{|V|}$ are learnable parameters, and $|V|$ is the vocabulary size.

The sequence discriminator $D_{t}$ is another recurrent network that distinguishes between real and generated poems. It assigns a score indicating the probability that an input sequence $y$ is real. During training, the poem generator aims to maximize the scores of generated poems, while the discriminator tries to maximize the scores of real poems and minimize those of generated ones.

For the sequence-to-image mapping, we construct a LSTM encoder, which has random initial seed for output diversity, and an end-to-end image generator. Different from the image-to-text translation, The poem encoder in this reciprocal task employees a Bidirectional LSTM (BiLSTM) [39] that encodes poems to produce a group of hidden sequence features.

Next, the sentence translator takes the hidden states and maps them to sentence space to get the sentence feature. Formally, the hidden states of poetry encoder are computed by:

where $t$ donates the sequence number, $e$ donates the text embedding, $\overrightarrow{h_{t}^{i}}$ donates the forward hidden sequence features and $\overleftarrow{h_{t}^{i}}$ donates the backward ones. To capture the global semantic information in the poem, we apply mean pooling over the hidden states to obtain a context vector.

The painting generator $G_{p}$ takes the context vector $c$ as input and synthesizes a high-resolution painting $\hat{x}=G_{p}(c)$ through a series of upsampling and convolutional layers. Following [25, 26], we employ a multi-stage generation process where the generator is decomposed into several sub-networks that progressively increase the resolution of the output image.

The painting discriminator $D_{p}$ is a convolutional network that assesses the realism of generated paintings. It outputs a matrix of scores $\mathbf{S}=D_{p}(\hat{x})$, where each element $S_{ij}$ represents the probability that the patch centered at location $(i,j)$ in the input image is real.

The adversarial loss for the painting generator is the mean of the pixel-wise BCE losses between the discriminator scores and an all-ones matrix:

where $W$ and $H$ are the width and height of the discriminator scores.

The discriminator loss summarizes the BCE losses for real and generated paintings:

By alternating between minimizing the generator loss and the discriminator loss, the model learns to generate paintings that are convincing to the discriminator.

The painting encoder $E_{p}$ is a pretrained ResNet-50 on ImageNet. It extracts a 2048-dimensional feature vector from the input painting. The poem encoder $E_{t}$ is a bidirectional LSTM with a hidden size of 512. It takes the character sequence of the poem as input and outputs the final hidden states as the poem representation.

The painting generator $G_{p}$ is a stack of six up-sampling and convolutional layers that progressively increase the resolution of the feature map to generate a 256$\times$512 painting. The poem generator $G_{t}$ is an LSTM decoder with a hidden size of 512. It takes the concatenation of the painting feature and the previous character embedding as input and predicts the next character at each time step.

The painting discriminator $D_{p}$ selects PatchGAN [40] component that operates on patches of size 64$times$64 to classify whether they are from real or generated paintings. The poem discriminator $D_{t}$ is a bidirectional LSTM followed by a binary classification layer that predicts the poem’s authenticity.

We implement the proposed network using PyTorch 1.12.0 [41], an open-source deep learning framework, with CUDA 11.6 for GPU acceleration. The implementation runs on a high-performance server equipped with an Intel Xeon E5-2680 v4 CPU (2.40 GHz, 14 cores), 128 GB DDR4 RAM, and dual NVIDIA GeForce GTX 1080 Ti GPUs (11 GB VRAM each). The operating system is Ubuntu 18.04 LTS. For optimization, we employ the Adam algorithm [42] with hyperparameters $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\epsilon=10^{-6}$. The learning rate is initialized at $10^{-4}$ and follows a polynomial decay schedule with power $p=0.9$.

Empirical experiments determined a 1:5 ratio between supervised paired and unsupervised unpaired data. This ratio balances the benefits of direct supervision from paired samples with the diversity and generalization offered by unpaired data. We found that this ratio yielded the best performance across our evaluation metrics. For poem generation, we utilize top-$k$ sampling with $k=12$ and a softmax temperature of 0.6 during inference to introduce controlled randomness. The maximum poem length of 80 characters was chosen to accommodate the longest common form of classical Chinese poetry, the seven-character regulated verse, which typically consists of 56 characters. The additional characters allow for potential variations and ensure that the model can generate complete poems without truncation.

We use a 70%/15%/15% train/validation/test split of the CPDD dataset. The training set is used for model training, the validation set for hyperparameter tuning and early stopping, and the test set is held out entirely until final evaluation. All reported metrics, including DCE, are calculated on this held-out test set to ensure unbiased evaluation.

Evaluating the quality of generated poems and paintings in the context of artistic poem-painting translation is a challenging open problem. We employ both automatic metrics and human evaluation to comprehensively assess the generated results.

To validate our proposed metrics, we conducted a correlation study comparing them against human judgments and existing metrics. We evaluated 100 randomly selected poem-painting pairs from CPDD dataset.

Five expert evaluators rated each pair on a scale of 1-5 for quality, fluency, coherence, and diversity. Figure 6 presents the Pearson correlation coefficients between our proposed metrics, existing metrics, and the average human ratings for poem and painting generation tasks.

For poem generation, the results demonstrate that our proposed metrics generally exhibit stronger correlations with human judgments compared to existing metrics. MCE shows the strongest correlation with quality ratings (r = -0.95), outperforming BLEU (r = 0.75) and METEOR (r = 0.78). MTE exhibits the highest correlation with diversity ratings (r = -0.96), surpassing Perplexity (r = -0.88). METEOR performs best for coherence (r = 0.95), while BLEU shows the strongest correlation with fluency (r = 0.92).

For painting generation, our proposed DCE metric demonstrates exceptional performance, particularly in assessing semantic consistency (r = -0.97) and style adherence (r = -0.88). It also shows strong correlations with visual quality (r = -0.82) and diversity (r = -0.75). The FID metric, while effective, generally shows lower correlation values across all categories compared to DCE.

These findings suggest that the proposed metrics effectively capture nuanced aspects of both poetic and visual quality, as well as cross-modal semantic consistency in the context of classical Chinese poem-painting translation. The MCE and MTE metrics prove particularly valuable for assessing poem quality and diversity, respectively, while DCE demonstrates broad applicability across various aspects of painting evaluation.

We conduct ablation experiments on the poem-to-painting task to evaluate each component in the proposed model. Table II shows the performance when certain components are removed.

From the results, we observe that removing any of the key components leads to a performance drop, confirming their individual contributions. The cycle-consistency loss plays the most crucial role, as removing it leads to the largest degradation across all metrics. This demonstrates the importance of leveraging unpaired data via cycle-consistent training for improving both the artistic quality and semantic alignment of the generated poems and paintings.

The adversarial losses are also beneficial, contributing to the realism and stylistic adherence of the outputs. Ablating the adversarial losses results in lower scores, especially for painting generation. Both the paired and unpaired data are valuable for our model. While the paired data provides direct supervision for learning cross-modal correlation, the unpaired data offers a rich source of poetic and pictorial patterns that enhance the generalization and diversity of the outputs. Removing either leads to inferior performance.

We also evaluate the model with proposed Distribution Consistency Error metrics. The full model achieves the lowest DCE scores, indicating its superiority in generating semantically consistent poem-painting pairs. Ablating any of the components increases these error rates, further validating their effectiveness in our framework.

We compare our approach with the state-of-the-art methods, AttnGAN [24], StackGAN++ [26], MirrorGAN [27], PPGN [51] and Liu et al. [14], for text-to-image and image-to-text translation. For fair comparison, all models are trained on the same data splits and evaluated on the CPDD test set.

The quantitative evaluation results for poem generation are presented in Table III. Our full model achieves the highest scores across all metrics, outperforming state-of-the-art methods by a large margin. In particular, we obtain an absolute gain of 19.4% in F1-score and 17.2% in Precision over the best baseline, demonstrating the effectiveness of our semi-supervised cycle-consistent approach in generating high-quality poems that closely resemble the human references. The BLEU and METEOR results also highlight the improvement in n-gram overlap and semantic alignment between the generated and ground-truth poems. These results suggest that our model is able to capture the rich poetic expressions and visual-semantic mappings from the painting-poem pairs, while leveraging the additional diversity and linguistic knowledge from the unpaired datasets. Furthermore, we evaluate the models using the proposed Mean Top-$k$ Error metric to assess the diversity of the generated poems. As shown in Table III, our approach achieves the lowest MTE score, indicating its superiority in generating a diverse set of high-quality poems that capture the artistic style and semantic content of the input paintings.

Figure 7 shows some poem samples generated from the proposed model. The poems are fluent, coherent, and capture the artistic conception and emotional resonance typical of classical Chinese poetry. The vivid imagery, includes ”Lofty towers”, ”rustic temple”, ”cowherd boy”, and poetic devices like parallelism and metaphors are well-executed.

The proposed model generates various forms of classical Chinese poetry, including five-character quatrain, seven-character quatrain, five-character regulated verse, and seven-character regulated verse. We conducted extensive experiments on generating Chinese paintings from seven-character regulated verses achieving remarkable results. Figure 6 showcases examples of paintings generated from seven-character poems, demonstrating our model’s ability to visually interpret complex poetic imagery and emotions. The model’s architecture allows for easy adaptation to other poetic forms by adjusting output constraints during decoding.

The evaluation results for painting generation are presented in Table IV. Our approach achieves the lowest FID score, indicating that the generated paintings exhibit high visual fidelity and diversity comparable to real paintings from the CPDD dataset. We also obtain the highest genre classification accuracy, demonstrating the model’s ability to synthesize paintings that adhere to the artistic styles and content of classical Chinese art. Additionally, we employ the Distribution Consistency Error (DCE) metric to assess the inter-domain correlation between the generated paintings and poems. As shown in Table IV, our approach achieves the lowest DCE score, confirming its superiority in generating semantically aligned poem-painting pairs with consistent feature distributions across the visual and textual domains.

Figure 8 shows generated paintings from given seven-character poems and Figure 9 displays the results from five-character poems. The corresponding paintings are visually realistic and accurately depict the semantic content of the poems. Our model is able to generate pictorial elements that are highly relevant to the poems, such as the fisherman, boat, seagulls, and sunset glow in the first example. The painting styles also closely resemble those of classical Chinese landscape paintings.

In summary, quantitative and qualitative results demonstrate the proposed model is capable to generate poems and paintings are artistic and semantically aligned. The diversity of the generated outputs highlights the benefits of the proposed semi-supervised training scheme in learning generalizable cross-modal mappings.

For human evaluation, we recruit 10 professional artists to score 100 randomly sampled poem-painting pairs on a scale of 1 to 5 along three criteria: poeticness (whether the poem is coherent, fluent, and poetically pleasing), picturesqueness (whether the painting is artistic, visually appealing, and thematically relevant to classical Chinese art), and semantic consistency (whether the poem and painting are well-aligned in terms of content and artistic conception). We report the average score for each criterion.

The human evaluation results are shown in Table V. Our model receives the highest scores in all three aspects, indicating its superiority in generating poetic, picturesque, and semantically consistent poem-painting pairs. Notably, our approach obtains an average score of 4.32 for poeticness, significantly higher than the baselines. The generated poems are deemed highly fluent, coherent, and aesthetically pleasing by the human experts, closely resembling the style of classical Chinese poetry. For picturesqueness, our model also achieves a high score of 4.25, demonstrating its ability to create visually appealing and artistic paintings that capture the essence of traditional Chinese art. The artists praise the vividness, composition, and finesse of the generated paintings. In terms of semantic consistency, our model obtains a score of 4.18, showing a strong alignment between the generated poems and paintings. The human experts confirm that the visual content and artistic conception conveyed in the paintings accurately reflect the semantic meaning and emotional tone of the paired poems.

These human evaluation results validate the effectiveness of our approach in generating high-quality and coherent poem-painting pairs that are well-received by professional artists. The cycle-consistent training schema and adversarial losses contribute to the superior performance in terms of artistic style, linguistic fluency, and cross-modal semantic consistency.

We compare the training and inference time of our model with the baselines in Table VI. Our model achieves a good balance between performance and efficiency. The training time is relatively longer than some of the baselines due to the additional cycle-consistency training on unpaired data. However, this is compensated by the significant improvements in generation quality and cross-modal consistency.

For inference, the proposed model is quite efficient, taking only 0.28s to generate a poem from an input painting, and 1.79s vice versa. This is comparable to most of the baselines and much faster than existing methods which require multiple stages of refinement.