Investigating Conceptual Blending of a Diffusion Model
for Improving Nonword-to-Image Generation

Given an interpolated embedding between two text prompts describing concepts A and B, respectively, we measure how often a text-to-image diffusion model exhibits conceptual blending of the two concepts. This paper selects Stable Diffusion²²2Stable Diffusion-v1-4 on the model card: https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md (Accessed August 7, 2024) (Vision and at Ludwig Maximilian University of Munich, 2022) as a diffusion model which uses the CLIP³³3CLIP ViT-L/14 on the model card: https://github.com/openai/CLIP/blob/main/model-card.md (Accessed August 7, 2024) (Radford et al., 2021) text encoder to compute the CLIP last-hidden-space text embeddings. For each interpolated embedding, $N$ images are generated for analysis.

A list of 1,000 existing English nouns representing different concepts is prepared, referred to as EvalNouns1000 hereafter. These nouns are randomly taken from the MRC Psycholinguistic Database (Coltheart, 1981) with the restrictions of word imageability and frequency. Low-imageable and low-frequent words are filtered out to ensure that the concepts can be depicted clearly in images and that the words are not rare in everyday use (See supplementary materials for more details). Next, for two nouns (denoting concepts A and B) selected from EvalNouns1000, an interpolated embedding in the CLIP last-hidden-state space, $\mathbf{e}^{\mathrm{hidden}}$, is calculated as a linear interpolation between the text embeddings of the two nouns/concepts. In detail, for each pair of the embeddings $\mathbf{e}_{\mathrm{A}}^{\mathrm{hidden}}$ and $\mathbf{e}_{\mathrm{B}}^{\mathrm{hidden}}$ corresponding to concepts A and B, the interpolated embedding is calculated as

where $r$ denotes an interpolation ratio ranging between 0 and 1. We use the prompt ``a photo of a <WORD>'' for calculating embeddings $\mathbf{e}_{\mathrm{A}}^{\mathrm{hidden}}$ and $\mathbf{e}_{\mathrm{B}}^{\mathrm{hidden}}$ where ``<WORD>'' denotes each concept. 1,000 such pairs are created by randomly choosing two nouns from EvalNouns1000. The interpolation ratio for each pair is randomly assigned from 0.1 to 0.9 with the step size 0.1. For each pair, $N$ images are generated using the interpolated embedding.

Given an interpolated embedding in the pooled embedding space, the task is to estimate its last-hidden-state embedding that preserves the positional relationships with its neighbors with minimum loss. In this evaluation, we first prepare pairs of interpolated embeddings in both pooled and last-hidden-state embedding spaces. Equation (1) and its analogy to the pooled output space are used for calculating the interpolated embeddings in each space. For each pair, the last-hidden-state interpolated embedding is regarded as the ground truth for the pooled interpolated embedding. This evaluation does not use actual outputs of the nonword encoder as shown in the generalized framework because there is no direct way to prepare corresponding last-hidden-state embeddings.

The proposed method performs $k$-nearest neighbor search on 26,143 data samples in the pooled embedding space and uses the corresponding 26,143 samples for the estimation in the last-hidden-state embedding space. These samples, which we refer to as TrainWords26143, are taken from the existing study (Matsuhira et al., 2023b, a, 2024). TrainWords26143 consists of 26,143 words listed on the Spell Checker Oriented Word Lists (SCOWL)⁵⁵5http://wordlist.aspell.net/ (Accessed August 7, 2024). The previous work selected those words based on word frequency and pronunciation availability. In this paper, we create an embedding from each word using a prompt ``a photo of a <WORD>''. We also confirmed that our EvalNouns1000 is a subset of TrainWords26143. This evaluation tests the hyperparameter $k$ with different integers ranging from 1 to 1,000.

We compare the proposed method with the MLP-based method identical to the one used in the existing study (Matsuhira et al., 2023b, a, 2024). Their training data for the MLP used a set of 26,455 words. This set is almost identical to TrainWords26143, with the only difference being that it contains an additional 312 words for which no pronunciation was available. To increase the number of samples, they created three prompts for each word in the wordlist: ``<WORD>'', ``a photo of <WORD>'', and ``a photo of a <WORD>'', although this augmentation is not performed in our method.

As evaluation data, we use the 1,000 matching pairs created in Section 3, which are pairs of two words randomly selected from EvalNouns1000, to prepare 1,000 interpolated embeddings.

As a metric for the information loss, we compute the L2 distance between a ground-truth embedding and an estimated embedding in the last-hidden-state embedding space averaged over all samples, which we call an L2 error. To see how well the neighborhood relationships are preserved, we also report Spearman's rank correlation between the rankings of neighborhood embeddings within TrainWords26143 averaged over all samples. The rankings for ground-truth and estimated embeddings are obtained by searching their $\ell$ nearest-neighbor embeddings in the pooled and last-hidden-state embedding spaces, respectively. All the nearest-neighbor searches during this evaluation are performed on an L2 distance basis. The metric is measured under two different $\ell$s: 2 and 5 (See supplementary materials for results under more various settings). Before calculating these metrics, we flatten each last-hidden-state embedding shaped $77\times 768$ into a $59,136$ dimensional vector.

Image generation quality is measured using Fréchet Inception Distance (FID) (Heusel et al., 2017). This evaluation measures two FIDs: $\mathrm{FID_{Orig}}$ and $\mathrm{FID_{Inter}}$. Calculating the former uses images generated from text embeddings of real text prompts which should depict clear visual concepts. In contrast, calculating the latter uses images generated from ground-truth interpolated embeddings which can exhibit conceptual blending as confirmed in Section 3. As a reference for calculating these metrics, ten images are generated from each of the 1,000 text embeddings of EvalNouns1000 and the 1,000 ground-truth interpolated embeddings introduced above⁶⁶6We calculate FID using a Python package pytorch-fid: https://pypi.org/project/pytorch-fid/ (Accessed August 7, 2024) with a prompt.

Results are shown in Table 2. First, we can see that the L2 error of the proposed method was always lower than that of the comparative MLP-based method with a great margin. This demonstrates the strong advantage of our interpolation approach over the direct prediction approach. The rank correlations of distances within a few neighborhood samples $\ell$ also showed a large gain over MLP, while tended to decrease as $\ell$ increased (See more results in supplementary materials). This can be explained as the curse of dimensionality, where the L2 distances measured in high-dimensional spaces become less diverse. Yet, the higher correlations of the proposed method within small $\ell$s indicate that it preserved neighborhood relationships better than the comparative method.

Furthermore, the proposed method showed better scores for both FID metrics than the comparative method. This indicates that our more precise and accurate last-hidden-state embedding estimation has improved image generation quality, too. Notably, $\mathrm{FID_{Inter}}$ showed 1.86 point at minimum when $k=300$. This small value indicates that the generated images using the proposed method were almost identical to those generated using the ground-truth interpolated embeddings. Since we have confirmed in Section 3 that the ground-truth embeddings can yield BCD in up to 60% image generation results, it suggests that the proposed method with $k=200$ or $k=300$ can also yield conceptual blending in a similar frequency, which will be assessed more deeply in the next section.

As an encoder to compute CLIP pooled embeddings for nonwords, we retrain the NonwordCLIP-P (Matsuhira et al., 2023b, a, 2024) pronunciation encoder with a customized dataset. Since the original training dataset contained various sentences from image captioning datasets, the trained model tended to be biased on the word frequency in the dataset. For instance, in the case of the nonword ``calve'' (/”kæv/) which we assumed to have the two most similar-sounding words ``calf'' (/”kæf/) and ``cave'' (/”keIv/), its nonword embeddings were always encoded in a similar position to ``calf'' because the dataset contained ``calf'' more frequently than ``cave''.

To avoid this, we construct a dataset in which each word appears almost an equal number of times. The dataset consists of 5,496 highly-imageable and -frequent nouns and noun phrases created by combining the MRC Psycholinguistic Database (Coltheart, 1981), a Python package wordfreq (Speer et al., 2018), and an English lexical database WordNet (Miller, 1995) (See supplementary materials for more details). We augment the dataset twice using the two prompts ``<WORD>'' and ``a photo of a <WORD>'', resulting in the training data of 10,992 samples.

When generating images, we follow the same settings as the exiting study (Matsuhira et al., 2023b, a, 2024), adopting the pronunciation prompt /@ ”foU””toU ”2v @ <NONWORD>/ (corresponding to ``a photo of a <NONWORD>'').

BCG and MCG ratios introduced in Section 3 are measured to assess conceptual blending in nonword-to-image generation results. To calculate these, this evaluation lacks ground-truth concepts A and B since the nonword embeddings are not computed by interpolating the embeddings of two concepts. Hence, to prepare pseudo-concepts A and B, we search the top-two nearest-neighbor words for each nonword embedding in the CLIP pooled embedding space. For instance, if the top-two nearest-neighbor embeddings of the nonword embedding ``calve'' are ``calf'' and ``cave'', our metrics detect occurrences of conceptual blending of ``calf'' and ``cave'' in the image generation result for ``calve''.

To align the metric calculation condition to Section 3, the nearest neighbors are searched from words in EvalNouns1000, and the hyperparameters for the metrics are set as $N=10$ and $n=2$. When searching the second nearest-neighbor word, we exclude words that are too close to the first nearest-neighbor word from the candidates. This is to avoid concepts A and B being semantically too similar (e.g., ``bloom'' and ``blossom'') and mis-detecting the image generation of only concept A as the emergence of conceptual blending. The threshold to judge closeness is set as the first percentile of the distribution of the L2 distance between each of ${}_{1,000}C_{2}$ pairs of two samples in EvalNouns1000.

For nonwords, we use Sabbatino et al.'s 270 randomly created English nonwords (Sabbatino et al., 2022), among which our evaluation uses only 242 nonwords whose embeddings are not located in a close position to existing words. Specifically, filtering is performed to restrict nonwords to be located in positions where the ratio of the distances to the top-one and top-two nearest-neighbor words, which corresponds to the interpolation ratio in Section 3, is between 0.4 and 0.6. The others are excluded because, as confirmed in Table 1, they are less likely to yield conceptual blending and can disturb metrics.

The results are shown in Table 3. First, we confirmed that the proposed method yielded a maximum BCG ratio of 0.707 and an MCD ratio of 0.789 when $k$ was set to 1,000. This BCG ratio is much larger than the maximum value observed in Table 1 when the interpolation ratio was 0.5. The MCG and BCG ratios of the proposed method had another local maximum at $k=200$, where the previous evaluation suggested the most accurate embedding conversion in the L2 error. These results indicate that the accurate embedding space conversion method did increase the chance of conceptual blending, but also suggest other factors that control the emergence of the effect. This can also be deduced by seeing the comparative method which yielded better BCD and MCD ratios than the proposed method with $k=200$ while producing a larger L2 error in Table 2.

To seek insights into those factors, we next look at actual nonword-to-image generation results exhibiting conceptual blending. Figure 5 shows ten images for each of the three nonwords ``calve'' (/”kæv/), ``broin'' (/”b⁢rOIn/), and ``blour'' (/”blaU@⁢r/) generated using each method. According to the top-two nearest-neighbor words, the nonword ``calve'' has similar sounding words ``calf'' and ``cave'', ``broin'' has ``brain'' and ``bone'', and ``blour'' has ``flower'' and ``flour''. The figure indicates that all the methods can depict the blended concepts of the two similar-sounding words. Meanwhile, they mainly differed in image generation qualities and abstractness of the depicted objects, as indicated in the FID metrics in the previous evaluation. Most notably, the proposed method with $k=1,000$ tended to exaggerate the texture of visual concepts, making the images very abstract. Also, as especially observable in Figs. 5 and 5, the MLP-based method tended to lack details of objects compared to the proposed method. Such abstractness can have overrated the concept detection metrics, as our classifier used in the metrics is prone to misdetection for images containing abstract objects that are recognizable in various ways.

As for why the inaccuracy of the embedding conversion did not affect conceptual blending, the dimensionality of the CLIP last-hidden-state embedding space can be the key. Last-hidden-state embeddings of the CLIP model used in this paper have the shape $77\times 768$, where $77$ denotes the maximum token length of a transformer model and each dimension corresponds to each token of the input text prompt. In our experimental setup, the input prompt used to create the evaluation data was always restricted to ``a photo of a <WORD>''. This prompt was generally tokenized into less than ten tokens, suggesting that the dimensions of only the first less than ten tokens were more essential than the other dimensions.

Considering this, we calculate the dimension-wise L2 error metric in the same setting as Table 2, as shown in Fig. 6. As expected, the MLP-based approach yielded more information loss than the proposed method in most dimensions. However, the loss became comparable especially in the dimension of the first token corresponding to the [CLS] token, which usually represents the global feature of a last-hidden-state embedding. A similar trend can be seen in the position of the seventh token, where [EOS] (End Of Sentence) token typically falls. These results indicate that the embedding conversion accuracy in the dimension of [CLS] and [EOS] tokens would be the most responsible for the occurrence of conceptual blending in text-to-image diffusion models. Stable Diffusion could be referring dominantly to these two dimensions for determining concepts to blend in our experimental setup.