The Five-Dollar Model:
Generating Game Maps and Sprites from Sentence Embeddings

Recent developments with text-to-image generators in the AI community have spurred a renaissance of large generative models. Both open and closed source generators, such as DALL-E and StableDiffusion have demonstrated their versatility and range in image generation, from hyper-realistic photos to abstract artworks. However, to generate these quality images, these large models require large amounts of training data—both in text captions and paired images—large amounts of compute and training time, and large amounts of storage. Though these models boast incredible textual understanding and image fidelity, they cannot be used in every domain. For more specific tasks, such as creating assets like sprites, textures, or levels for video games, these models require extensive prompt engineering or fine tuning to meet domain-specific constraints.

Within procedural content generation (PCG) research, various smaller models have been proposed specifically to create assets. These models are much simpler in architectural design and include evolutionary models, GANs, VAEs, and others. However, most PCG models lack conditional prompting. Conditional prompting is a technique used to guide the generation process towards more specific artifacts using a provided condition, such as a class label or text description. A small-scale model that requires low amounts of training data and has direct text-to-image generation capabilities would benefit game developers considerably, as it would facilitate content creation and provide a greater degree of control over the output of the AI system.

We introduce the five-dollar model¹¹1A model so simple, one of the authors bet another $5 it wouldn’t work.—a text-to-image generator designed for low-dimensional, pixel-size images and categorical-based samples. We use this model to generate game levels (Figure 1), sprites (Figure 2), and emoji images (Figure 3). We find that despite the conceptual simplicity and comparatively small size of our model, it demonstrates surprising performance and language understanding in these domains. This generator is remarkably fast because of its small size and lack of iterative steps, making it suitable for real-time content generation. It is also easy to train because of the lack of adversarial dynamics (unlike GANs) and it can be trained using relatively small training sets. Because of the lack of denoising (unlike diffusion models), it is suitable for discrete and categorical content such as tile-based maps. Finally, the simplicity of the model makes it easier to study and potentially modify after training.

Procedural content generation in games is an active field of game AI research. This field includes algorithmically creating artificial samples for game content, such as levels, sprites, or narratives (Shaker, Togelius, and Nelson 2016). Previous works have used evolutionary algorithms (Charity and Togelius 2022), generative adversarial networks (GANs) (Volz et al. 2018), variational autoencoders (VAEs) (Sarkar and Cooper 2021), VQ-VAEs (Saravanan and Guzdial 2022), cellular automata (Earle et al. 2022), diffusion models (Xu et al. 2023), transformer models (Todd et al. 2023), and more.

Conditional generation allows for more control over the output of the black-box neural network models by providing auxiliary information, called the condition. Conditional information can come in many forms. Class labels (Mirza and Osindero 2014), text descriptions (Reed et al. 2016), and other images (Dumoulin, Shlens, and Kudlur 2017) have been used as conditional inputs to guide image-generation models towards a specific goal. Works done by Van den Oord (Van den Oord et al. 2016), Kang (Kang and Park 2020), Mirza (Mirza and Osindero 2014), Perez (Perez et al. 2017), and Isola (Isola et al. 2017) with conditional generators on various domains such as face and image generation demonstrate the effectiveness of control over the final output when using conditional inputs all the while retaining quality.

Text-to-Image generation employs natural language image descriptions as a means to condition generative image models. This class of models aims to generate high quality images that semantically align with the provided input prompt. With the rise of conditional diffusion (Sinha et al. 2021) and Stable Diffusion models (Ramesh et al. 2022), text-to-image generation has seen a huge increase in popularity. Current research in this domain focuses largely on the zero-shot capabilities of such models — the ability for them to generalize to unseen data.

Procedural Content systems are most often designed for a specific game, where zero-shot generalization is not a primary concern. With this paper, we aim to explore domain-specific text to image generation in the context of digital game art. In contrast to state of the art systems, which are typically trained on large scale datasets of real world RGB images, we train on low dimensional categorical data.

We create various experiments to evaluate the language understanding and generative abilities of the five-dollar model in each domain.

First, the latent space of the model is investigated using interpolation, also known as “walking the latent space” (Radford, Metz, and Chintala 2015). A smooth latent space should result in a gradual transition of the intermediate outputs between the two associated images. We manually select two points in the dataset with either similar semantic meanings, or similar image representations for these experiments.

Secondly, to evaluate the natural language understanding of our model, we use latent vector arithmetic. Using arithmetic operations on the encoded text vectors of two similar images, we attempt to isolate vectors which represent important semantic concepts within the image domain. For example, by subtracting “neutral face” from “angry face” in the emoji domain, we obtain a vector that represents “anger”. We can then visualize adding this vector to other text prompts, to see whether the model encodes an intuitive understanding of these concepts.

We lay out the entire training pipeline in 4 steps: (1) Alternate Label Creation with GPT-4 (2) Data Preparation, (3) Data Augmentation, (4) Model training. Step (1), the alternate label step consists of generating the alternate labels from GPT-4 in order to increase the base size of the dataset. Step (2), the data preparation step, includes quantization of images into one-hot encoded representation and encoding of text labels using the embedding model. Step (3), applying the augmentations mentioned above such as MixUp and noise augmentations. Finally, step (4), the model training stage, which in our case included a grid search for the best performance in each domain.

At inference time the only models used are the pretrained embedding model and the five-dollar model, completely foregoing of steps 1 and 3. Large models, such as GPT-4, are only used during the alternate label creation step. Thus, GPT-4 gives no increase in computational resources required to use this model in downstream applications.

To fully understand the capabilities of these models we aimed to optimize the following hyperparameters through a grid search: the noise input vector’s dimension, the filter counts, sizes of the kernels, the number of residual blocks, and the conditioning type³³3The three options for conditioning types are concatenation based (depicted in Figure 4), Conditional Instance Normalization (CIN layers after every residual block), or Feature-wise Linear Modulation (FILM layers after every residual block).. The models were evaluated based on validation accuracy within their domains with validation sets consisting of withheld human-made labels and a matching pixel map, sprite, or emoji respectively for each of the domains.

While a correlation between validation accuracy and output quality is noted, the augmentation strategies used on the dataset introduced some information leakage between the training and validation sets. To combat this, we evaluate the top performing models in two ways. ⁴⁴4When deciding between two models with similar performance, the model with the lower number of parameters was selected.

First, we subjectively evaluate the top models’ performance on various sets of unseen labels. These were designed to test the semantic understanding of the models, such as applying a different color to a learned outline i.e.: duck with a pink shirt as the unseen sprite, with duck, pink, and shirt all having independent examples in the training set. More examples of unseen prompts can be found in Figure 13 in the Appendix.

Secondly, we use CLIP Score as a measure of the text-image alignment. Contrastive Language-Image Pre-Training (CLIP) models embed similar text labels and images into an overlapping embedding space, aiming to maximize the cosine similarity of the embedded vectors for corresponding text-image pairs (Radford et al. 2021). We use a pretrained clip-vit-base-patch32 as the CLIP model in this paper. A scaled version of this similarity metric, called the CLIP Score, has a high correlation with human judgements (Hessel et al. 2021). We compute the Clip Score as the scaled cosine similarity, with a scaling factor of $2.5$.

We utilize this metric as an indicator of model performance without the need for human review. In contrast to some state-of-the-art generative models, our model does not include direct optimization for text-image alignment when training, maintaining the lightweight characteristics of the model. However, CLIP Score has limitations as a metric for our model evaluation. Due to the low dimensionality of our data, the domains of our images diverge significantly from the datasets used to train CLIP. Therefore, knowing CLIP performs well on zero shot tasks, and CLIP Score demonstrates its ability to reason about non-photographic art, we perform experiments to validate its use in our domains.

For each domain, we generate a fake dataset by pairing real captions with randomly tiled images⁵⁵5These were generated through a simple RandInt() function in the needed shape to match the real data.. We then compare the average CLIP Score of the real dataset vs the fake, random-tile dataset. We determine the validity of this metric by comparing these two scores: A significantly higher score on the real dataset vs the random dataset points to some semantic understanding of the pretrained CLIP model in our domains.

Lacking performance was noted when prompting CLIP with only the raw text input given to the generator models. Therefore, to gain a more representative metric we prepend the text with a dataset specific preprompt to guide the CLIP model towards our domain-specific task. The preprompts for each domain can be found in Figure 14 in the Appendix.

Now that we have target CLIP Score for our generators and after finding candidate hyperparameters via grid search, we compute CLIP Score of each model using a set of unseen labels. Table 2 shows the Clip Scores of the top standard and non-standard models in each domain⁶⁶6”Standard” refers to models without added CIN or FiLM conditioning.

We find that the five-dollar Model was able to generate aesthetically pleasing images that were semantically consistent with the input prompt in all 3 domains. Though our models suffered from some overfitting, particularly in the smallest sprite dataset, augmentation strategies were able to significantly improve generalization and metric performance.

The 3 different datasets resulted in five-dollar models with varying properties. The map generator seemed to perform the best when given descriptive strings. This dataset consisted of sentence-long descriptions of maps, and we found the best results came from clearly describing scenes to generate.

The map model also demonstrated a good understanding of the tileset. Text prompts that specify tiles, such as ”trees”, ”rocks” and ”water” were consistently accurate. We experiment with generating island maps with different features, depicted in Figure 6. Additionally, providing a single word text prompt such as ”rocks” generated maps entirely covered by that tile. We utilize and explore this property in the vector arithmetic experiments shown in Figure 7.

The sprite generator was most effective in generating variations of existing characters, but understandably struggled to depict entirely original characters. This was the smallest of our datasets with only 100 samples, and these models often struggled with overfitting and memorization. The structure of this dataset (including similar characters with slight variations, such as color) gave this model the best understanding of color. We explore this in the vector arithmetic experiments shown in Figure 8.

The emoji generator had the poorest performance in terms of generalization and generation of novel images. This dataset had the most diversity of images and subjects depicted. While emoji faces have a consistent structure and style, other categories like animals and flowers vary dramatically. While the model demonstrated good understanding of semantically similar descriptions (ex: Angry, Mad, or Furious face), it struggled to generate emoji faces depicting unseen emotions. Experiments on this model are shown in Figure 9. We tested models trained on only emoji faces, but did not notice much improvement.

We find through interpolation experiments that the embedding space in all 3 domains is smooth, and results in smooth transitions between interpolated images. Additionally, we are able to perform vector arithmetic within the embedding space. This is demonstrated in Figures 7, 8, and 9.

We pursued this research topic as a means to have an alternative to generative adversarial networks and diffusion models for generating small content fast, especially when this content may not naturally be an RGB image. We especially focused on the lightweight aspect of the model as smaller models are more accessible. Having smaller models enables applications that larger models would not allow, such as real-time local generation. Exploring the model under the constraint of being much smaller than state-of-the-art alternatives also allows for easier reproducibility, iterative testing, and accessibility.

Viewing these results through the lens of procedural game content generation it can be easy to imagine many possible use cases. Players could decide what the map, the characters, or any other asset looks like for infinite replayability. In the extreme, this could be done frame-to-frame with this new model’s lightweight capabilities. This would allow for variability not just between user-prompted, generative playthroughs but to have individual characters with individual sprites, on tailor-made maps from any device all in real time.

That being said, it is important to realize the limitations of these new models. While this work demonstrates the capabilities of lightweight aesthetic generation in a novel, easily interpretable model, it may be the case that these architectures have been pushed close to their limit and could only have a limited amount of room to improve. The zero-shot capabilities of these models are also very limited as they are only able to generate pixel images related in some way to the training data, though possible future works could build on this issue to resolve it through clever prompting.

This work has demonstrated the effectiveness of our simple, lightweight architecture for low-dimensional image generation with possible uses from video game maps to pixel art sprite generation. It also demonstrates the applicability of smaller models for specific use cases, even with limited data through the numerous augmentation methods discussed.

Though we find that GPT-generated alternate labels and MixUp augmentations can improve the performance of our model in these small datasets, there are many more augmentation strategies we did not explore. NLP augmentation methods such as synonym replacement and back-translation may further improve the language understanding and generalization capabilities of our models. There may also be useful computer vision augmentation methods. Though we experimented with horizontal flips and image noising, these methods often introduced artifacts into generated images. Due to the categorical representation of our images (rather than the more common RGB), and extremely small image dimensions, many common image augmentation strategies are difficult or impossible to apply to our datasets.

We do not explore the use of attention mechanisms in our networks. Cross attention has shown incredible potential in text-to-image models, as demonstrated by models such as CAGAN (Schulze, Yaman, and Waibel 2021) and AttnGAN (Xu et al. 2018). Similar to CIN and FiLM, while attention may improve our performance, it also adds a significant amount of size and computational complexity to our models. We leave exploration of attention for categorical pixel image generation to future work.