Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works

We examined six primary sources known for quality HCI research: the ACM CHI Conference on Human Factors in Computing Systems (CHI); the ACM Symposium on User Interface Software and Technology (UIST); the ACM Conference on Intelligent User Interfaces (IUI); the ACM SIGCHI Conference on Designing Interactive Systems (DIS); the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW); the ACM Conference on Creativity & Cognition (C&C). AI venues were excluded since we wanted to focus on the human-side understanding of the research. Particularly, we wanted to understand how humans can utilize technology to support their work, rather than quantitative evaluation or advancement of state-of-the-art AI technologies. We constrained the publication period to the past five years (i.e., from 2018 to 2022) to reflect recent trend of the research (Speicher et al., 2019).

Since it has been just a year since DALL-E was released, there exists only a handful of HCI research that focused on LTGMs directly. Instead, we investigated papers on generative models, considering that LTGM is another type of generative model trained at scale (Bommasani et al., 2021). For a thorough inspection of all the relevant publications in the above venues, we tried to be inclusive as much as possible. Namely, rather than finding papers that contained the exact term ‘generative model’, we tried to embrace all existing papers which include extended forms of search keywords such as ‘generate’, ‘generation’, ‘generative’, and ‘generating’ in their title or abstract.

After running an exhaustive search, we set 6 exclusion criteria, referencing previous notions (Scuri et al., 2022; Baykal et al., 2020), to eliminate papers that do not align with the goal of this research.

• EC 1:

  The common use of the term ‘generate’, ‘generation’, ‘generative’, and ‘generating’ do not align with our research direction. For example, we removed if the term ‘generative’ was used as a general adjective, not used to mean generative models. This criteria was required as we gathered papers inclusively in round 1.

• EC 2:

  We chose system/application papers that used generative models to support their tasks. In this regard, qualitative/study papers (Oh et al., 2020; Ross et al., 2021) were excluded.

• EC 3:

  We chose papers that use or study neural network-based generative models such as GAN, VAE, LSTM, GPT-3, and DALL-E. For instance, the use of conventional generative models like Gaussian mixture models was excluded.

• EC 4:

  Since we are interested in the impact of generative models to humans, excluded are the technical papers which adopted generative models for data augmentation (Maghoumi et al., 2021; Li et al., 2021a), synthesizing (Janveja et al., 2020), and complement (Wang et al., 2021b).

• EC 5:

  We excluded doctoral dissertation, survey paper, book, book chapter, and demo papers.

• EC 6:

  Lastly, we excluded the paper published by the same author(s) where the content is not different significantly.

Our strategy to perform the literature review was to collect papers broadly and then narrow down the scope for careful examination (Frich et al., 2019). Therefore, in round 1, we collected a total of 1,494 papers from the 6 venues (i.e., CHI, UIST, IUI, DIS, CSCW, C&C) using the search terms (i.e., generate, generation, generative, and generating) on a popular academic database, ACM Digital Library³³3https://dl.acm.org. In round 2, papers with the exact same title and content were removed (i.e., EC 6), which left 1,429 papers. In round 3, we read the title and abstract of the remained ones, and checked EC 1—5 to exclude papers that do not fit in the goal of our research. Lastly, in round 4, we read all the articles that passed all the previous rounds (i.e., 64 papers), and additionally added in 8 papers that were not found during the above process. Thus, we finalized our collection as a set of 72 papers. Table 1 shows the precise number of papers in each round.

We noticed that generative models were used to help diverse types of users. In total, we found 9 codes: artists, children/students, designers, engineers, general public/unspecified, the disabled, researchers, writers, and etc. The vast majority of the publications targeted general public or they simply unspecified the user (45.8%, 33 publications). Next, 13 publications (18.1%) targeted designer as their user. Different types of designers were included throughout the papers including fashion designers (Padiyath and Magerko, 2021), graphic designers (Ueno and Satoh, 2021), poster designers (Guo et al., 2021), and so on. We found 7 publications (9.7%) which targeted writers like novelists (Chung et al., 2022b), amateur writers (Yuan et al., 2022), and poets (Gero and Chilton, 2019). There were 6 publications (8.3%) that targeted researchers such as AI researchers, UX researchers, and graduate students. We identified 5 publications (6.9%) which targeted artists like digital artists (Yurman and Reddy, 2022), 3D modelers (Abdrashitov et al., 2020), and webtoon authors (Ko et al., 2022). There were 3, 2, and 1 publications (4.2%, 2.8%, and 1.4%) for children/students, engineers, and the disabled, respectively. Lastly, 2 publications (2.8%) were classified as etc. which includes domain experts in gesture generation (Yoon et al., 2021) and image restoration (Weber et al., 2020).

A total of 8 codes were identified in the review process. Specifically, writing was the main task in 13 publications (18.1%). This includes different writing scenarios such as story writing (Biermann et al., 2022; Yuan et al., 2022), email writing (Buschek et al., 2021; Liu et al., 2022a), and scientific writing (Gero et al., 2022). Everyday-work was the target in 11 publications (15.3%) such as podcast listening (Laban et al., 2022), video captioning (Yuksel et al., 2020), and conversation (Huber et al., 2018) which can be encountered in our everyday life. There were 13 publications (18.1%) targeting design, which captured various sub-domains such as fashion design (Padiyath and Magerko, 2021), furniture design (Urban Davis et al., 2021), UI/UX design (Gajjar et al., 2021; Ang and Lim, 2021; Liu et al., 2018; Wang et al., 2021a; Mozaffari et al., 2022), and graphic design (Ueno and Satoh, 2021). We identified 10 publications (13.9%) which targeted drawing including illustration (Liu et al., 2022b), computational drawing (Wang et al., 2020), human-AI co-drawing (Fan et al., 2019; Oh et al., 2018; Karimi et al., 2020b), and webtoon drawing (Ko et al., 2022). There were 9 (12.5%), 3 (4.2%), and 2 (2.8%) publications in composition, programming, and education, respectively. Lastly, 11 publications were grouped as etc. (15.3%) which contained tasks like tactile map generation (Hofmann et al., 2022) and choosing photo-realistic images (Zhang and Banovic, 2021).

We recognized five codes as the role of the generative models, which are automation, co-work, exploration, mediation, and representation. Unlike other themes, multiple codes were allocated to a single publication if needed, since we found it inefficient to build the structure to become mutually exclusive and exhaustive at the same time. However, no more than two codes were simultaneously assigned to a single paper. Our codes came out differently from a previous work that delved into creativity support tools (Chung et al., 2021), likely because we focused only on papers with generative models.

The automation means that the generative model was used to automate or replace human efforts to perform a specific task without any user interface (40.3%, 29 publications). For example, Singh et al. (Mittal et al., 2020) introduced a prototype to create a personalized emoji from a rough drawing where the user’s level of intervention was extremely restricted. The co-work indicates that the generative model was integrated into an interactive system to help perform a task in a human-in-the-loop or a mixed-initiative approach (41.7%, 30 publications). For example, Calliope (Bougueng Tchemeube et al., 2022) provided a user interface containing several levels of granularity to manipulate parameters (e.g., tempo) in generating MIDI files. Although both automation and co-work support the users in doing their tasks, the main difference lies in whether there exists any user interface with which humans can interfere.

The exploration signifies that the generative model was used to help users’ creative thinking process so they could get inspired to produce novel ideas (44.4%, 32 publications). For instance, Fede et al. presented The Idea Machine (Di Fede et al., 2022) using large language models to empower user’s idea generation in writing. The mediation means that the generative model was used to facilitate or arbitrating in communication between the concerned parties (13.9%, 10 publications). EmoBalloon (Aoki et al., 2022) used generative models to make emotional speech balloons to decrease the gap of emotional arousal between the sender and receiver of a text message. Lastly, the representation means that the generative model was used to embed information in a latent space for downstream tasks (12.5%, 9 publications). For example, Screen2Vec (Li et al., 2021c) showed semantic embeddings of GUI screens and components.

Among 28 interviewees, 12 acknowledged that LTGMs can be a new image reference search tool. They pointed out that referencing visual materials (e.g., images and videos) is a crucial part of their creative work. They mainly use references to 1) learn by observing what and how others create and 2) get inspirations for new ideas. In general, it helps to realize their imagination into the world. However, there were some exceptions as well. For instance, P21 (sculptor) uses references to check that the work she plans to make does not yet exist. References that are irrelevant to their field of domain are also utilized by 7 visual artists, suggesting that they help eliciting novel ideas. P20 (industrial designer) said:

P20, Industrial designer: If I were to design a speaker, I do not just find a bunch of speaker images. Rather, I find various images ranging from fine arts to architecture, and gather them all together as a collection. Next, the images are classified into several themes following the client’s design requirements to make several design prototypes.

All the reference seekers complimented that LTGMs are fast, convenient, and can make a large number of high-quality images that are unique and different from one another. Specifically, P9 (jewelry designer) acknowledged that LTGMs seem to be useful in the early stages like planning where a huge amount of images are needed for ideation. Also, P23 (contemporary art artist) argued that she felt more passive when using Google, but felt more active with LTGMs since they gave more curated samples using functionalities that require more human engagement (e.g., selecting regions with brushing). Despite all the aforementioned advantages and distinct features, 10 of the 12 reference seekers said LTGMs would not become a tool to shift their working paradigm, rather just another option for reference searching. P25 (VFX artist) stated:

P25, VFX artist: The advantage (of LTGMs) is that it can generate any images with several detailed conditions. Although Google is big, it cannot provide if the image is not contained in its database… Frankly speaking, I think people will use LTGMs, but there would not be a huge difference. Currently, if there are 100 people, around 90 use Pinterest. However, in the future, some of them will say ‘I use LTGMs’. That is all.

The detailed codes of User, Task, and Role are the following:

• •

  User: Visual reference seeker

• •

  Task: Ideation

• •

  Role: Exploration

Through the interviews, we confirmed that 20 visual artists acknowledged LTGMs would be beneficial in real-time visual communication for those who 1) work in a company adopting a top-down decision-making procedure, 2) commission to provide professional services to clients, and 3) interact with varying artists in different fields. Specifically, 11 visual artists who communicate with their boss back and forth in verifying and improving their work wanted to leverage LTGMs for a fast prototyping. They mentioned that the concept of art delivered from their boss is often vague, which requires several exploration in varying directions, especially in early stages. Therefore, they have to make many drafts as alternatives for confirmation before finalizing the concept, with most of the drafts ending up abandoned. The visual artists want to save their time and energy by taking advantage of LTGMs for such dissipated work. For example, P1 (webtoon author) stated that using LTGMs could decrease the time consumed in communication between her and her manager by performing a real-time revision of the high-level concepts. Specifically, she said she wants to make several images and let the manager make the concept as an ensemble by extracting and combing parts from the generated images. Similarly, P16 (concept artist) pointed out that LTGMs could guide the working direction in a case where the boss does not have a concrete goal in mind. P16 said:

P16, Concept artist (movie, game): While some art directors indicate a specific direction, others do not have one, but just want me to give some ideas. In this case, I have to show them many drafts to see their reaction and decide what to focus on and what not to… We usually work as a group of concept artists, and the director want us to draw the concept art with the same topic, respectively. If one is picked, then it is branched out further. We could use LTGMs to generate starting pieces for conversation in the initial discussion we can work based upon.

Moreover, 14 visual artists mentioned that LTGMs would help facilitate communication between the visual artists and their clients. Many visual artists commission artworks, offering professional services. In doing so, they have to meet the client’s requirements, which can often be ambiguous and even paradoxical at times. P8 (product designer) mentioned that once a client wanted a design that has both a ‘cold’ and ‘warm’ mood at the same time. She said the best way to communicate with the clients is often using visual materials where LTGMs can be a viable means. Similarly, P4 (motion graphic designer) stated that LTGMs would make it much easier to persuade clients as it enables fast prototyping. She added that communication often requires an iterative process, but she would be able to create a bunch of rough images in a short time using LTGMs. Moreover, when collaborating with many coworkers, it is essential to narrow down the gap between their thoughts on the final artifact. P14 (product designer) said LTGMs could decrease the time needed to reach a consensus:

P14, Product designer (character animation): Our work requires communication (between workers) as mandatory, as the design of a product can be pleasing only to me (but not to others). By taking others’ feedback, we calibrate the direction on how to revise it. Thus, we try to take comments as much as we can before embarking on real work. LTGMs can decrease the time for the overall communication process.

Lastly, 2 visual artists mentioned that LTGMs could be useful for those who interact with people across multiple domains. In detail, P24 (projection designer) pointed out that the communications with dancers and musicians are different as it requires a domain-specific understanding. He ascribed the fundamental reason to the inconsistent connection between text and visual materials:

P24, Projection designer: The language used by dancers, projection designers, and musicians is all very different. For example, dancers express abstract words, such as ‘joy’ through their work, but projection designers often find it hard to do the same with videos (as they do not have a logical connection). Therefore, we communicate via a bunch of images and ask them which one is visually close to your thoughts on ‘joy’, for example. LTGMs can be of use in such cases to narrow down (the search space) and reach an agreement.

The detailed codes of User, Task, and Role are the following:

• •

  User: Visual communicator; Client/Contractor in visual art work

• •

  Task: Real-time visual communication

• •

  Role: Mediation

In the study, 14 visual artists said that LTGMs could help them try unconventional things and think out of the box. Interestingly, 8 visual artists directly mentioned that humans have a bias in creating art, but LTGMs do not seem to have one. This is a finding that contradicts a previous research which revealed that people have a preconception on AI that it generates biased artifact (Bennett et al., 2021). The visual artists pointed out that humans have preferences, which can lead them to certain directions in performing creative works. They wanted to use LTGMs for attempting what they have never tried, deviated from their comfort zones. P17 (artist) said:

P17, Artist (video, painting): The advantage of LTGMs is that they do not have a bias (in generating art). Thus, it can generate any combination I make in a totally unexpected way. My personal knowledge and skills will be reflected to the work in previous approach, for example, the working style I stick to so far. However, the output of LTGMs is random, which is good in that it can recommend a new style of work.

Aside from personal preferences, 3 out of 8 visual artists mentioned that there are more practical reasons that prevents visual artists from taking unusual approaches. They said that it necessitates a lot of time and energy but often results in an unsatisfactory outcome. However, they praised that LTGMs encourage visual artists to take adventurous approaches, as it shows what they have in mind in a very small amount of time. P6 (illustrator and picture book artist) said:

P6, Illustrator and picture book artist: Although I desire to draw a night sky with purple color and a yellow house (as I think they look good overall), in reality, they might not harmonize with each other. Also, I need to think about the size of house and all the other details… Until the best one is picked, it takes a lot of time. Along with such a long procedure, I cannot find most of what I wanted at the first place from the final result. LTGMs can be a viable solution to make my imagination to real in short time.

The detailed codes of User, Task, and Role are the following:

• •

  User: Unconventional visual artist

• •

  Task: Unbiased prototyping

• •

  Role: Automation

A total of 9 visual artists pointed out that it would be helpful for those 1) who have low expertise in handling domain-specific art creation tools, and 2) who have little knowledge in related art fields, because LTGMs can help them generate low-fidelity prototypes. Since new software comes out every year, visual artists need to learn new skills constantly to not fall behind. However, the novices have difficulties in knowledge acquisition because it takes huge amount of time. For example, P7 (artist) did not use any software but performed manually for making a blueprint of artwork because she was too busy during working hours, although she knew it was inefficient. She wanted to leverage LTGMs in her work, as they can handle time-consuming tasks like image synthesis using simple interactions (e.g., brushing) and text prompts. Also, they said working in the visual art domain often requires knowledge and competencies across multiple fields. For example, since the company which P4 (motion graphic designer) works for only had two designers, she had to undertake all design-related tasks, although she only had experience in 2D motion graphics. P4 stated that LTGMs can help visual artists to embrace tasks beyond their capacity. Although the quality of created prototypes would be low, it would be of great help, when the outcome is needed urgently.

The detailed codes of User, Task, and Role are the following:

• •

  User: Novice visual artist

• •

  Task: Low-fidelity prototyping

• •

  Role: Automation

We found that LTGMs could be a tool to create artworks as well as justification for them in the fine art domains. Specifically, P12 (artist) wanted to use LTGMs in creating his painting because it is more credible once AI guarantees the generated results on a subjective concepts. Also, he mentioned artists prefer to take advantage of new technologies, so there can arise a new artistic paradigm called ”an era of AI art” that is a complete prosperity of AI-generated contents:

P12, Artist (video, painting, sculpture): As my topic of art is ugly people, I would like to generate 100 couples of men and women using LTGMs. Then, I would synthesize each pair to generate 50 pairs as their descendants. Through iterations, there would be the final one who is the epitome of an ugly person.
Interviewer: We can do that using Google. What is different from using LTGMs in doing that work?
P12, Artist (video, painting, sculpture): It is hard to find many different images for a certain text prompt in Google. Moreover, if I find images of ugly person on Google, others might disagree as it is based on subjective perception. However, I can justify my claim on the artwork if the generated images are made by AI. Who would argue with that if AI says so?

The detailed codes of User, Task, and Role are the following:

• •

  User: Unconventional artist

• •

  Task: Creating and justifying generated artworks

• •

  Role: Automation

A total of 5 visual artists said LTGMs are logical machines that can only create predictable images, when what they wanted were random explorations. Because getting inspiration comes from unexpected events, they wanted to find images that do not exactly match with their input text prompt. P22 (editorial designer and logo designer) mentioned that there exists a logical connection between the images generated by LTGMs and the input text prompt, making it impossible to elicit any novel ideas further. Moreover, P11 (spatial designer) mentioned that it is important to add such randomness in work, as it looks tedious and outdated if the concept is used directly. P11 said:

P11, Spatial designer: We work based on the concept and clothing to display. Then, we plan how to decorate the place considering the fabric and the atmosphere of the season… Although our concept is a flower road, for example, we cannot just display a flower road; that feels old-fashioned… That is why we find references from diverse sources.

Moreover, they pointed out that LTGMs cannot create images that embody philosophical meanings, storytelling, or sophisticated interpretation. For instance, P8 (product designer), who studied both fine arts (painting) and design, mentioned that the images generated by LTGMs are not interesting from the painter’s perspective. She was even able to predict what the outcome image of LTGMs would be in advance. In specific, she said LTGMs would create an ordinary red apple when she type in ‘why is an apple red?’, and the outcome was the same as her expectation (Figure 3-A). However, what she wanted was an image conceptualizing the philosophical reasoning that people can take to a further contemplation.

A total of 11 visual artists doubted whether LTGMs could generate images that require domain-specific understanding. For example, visual artists thought LTGMs are hard to reflect user experience and practicality. For example, P3 (Web/UI/UX designer) was surprised to see the generated image of mobile UI screen, but was uncertain about LTGMs’ capabilities to consider the user flow. He worried that it will just visualize the given conditions without considering the user experience (Figure 3-B). P18 (fashion designer) and P27 (architect) gave similar opinions. They acknowledged that LTGMs can generate rough sketches of artifacts (i.e., clothing and building), but it cannot handle all the detailed requirements of the design. Specifically, P27 mentioned that architects have to consider practical issues like whether it can actually be constructed by humans while designing the architectural layout. However, she thought LTGMs would not include realistic considerations in the layouts (Figure 3-C).

Moreover, the visual artists wanted to incorporate their identities into the artifacts. For example, P10 (artist) stated how visual artists would not be able to add their own identity to the artifact if LTGMs managed all the work. Similarly, P17 (artist) wished for customization of the generated images to reflect her own style since she worried that they would contain some sort of characteristics of LTGMs, so that they do not seem something special, but stale artwork.

A total of 9 visual artists pointed out that LTGMs cannot generate novel images due to their dependence on text prompting. We found two main reasons for this. First, since LTGMs use text prompts, they needed to have a word or phrase to describe what users want to create. However, when creating something that is entirely new, there would be no word for it yet, which results in a paradoxical situation. For example, P8 (product designer) who designs a medical app tried to generate images using the text prompt ‘Mohei’, which is a provisional brand name for her own product. The original meaning of Mohei was an abbreviation of ‘sunshine on the corner’, but the output image was a totally random image such as a dumpling (Figure 3-D). Similarly, P10 (artist) who draws abstract art mentioned that it is impossible to find pertinent text prompt for realizing his thought. This is because he prefers to seek a possibility that comes from consecutive random brush strokes, which is an artistic technique called impasto. His art becomes completed through a ceaseless interaction with the canvas. Therefore, he found it is hard to describe the overall art-making process in texts. P10 stated:

P10, Artist (abstract art): I worry about the fundamental limitation of text prompting. In the planning, I do not use text, but find what I want via painting. For instance, a word ‘dark’ can represent several imagery with varying levels and combinations of darkness. There can be subtle differences between them, and I love to focus on the gap from which I can discover such representation.
Interviewer: What is the problem of describing it into a text?
P10, Artist (abstract art): The delicate feeling will volatilize while translating it. As I need to take time to choose which one is a right word, the first imagery came to my mind will go away.

Second, P27 (architect) mentioned that it would be really convenient if all the existing styles are trained on the LTGMs so they can create any of them using a simple text prompt. However, she also admitted that it would limit the styles of generated images within the boundary of trained samples. Thus, LTGMs would rather operate as something confining the visual artists’ imagination. P27 said:

P27, Architect: It would be really convenient if all existing floor plans are trained on the LTGMs, and they can generate whatever I want… However, if that is possible, I believe they can rather confine the human’s imagination. This is because images that can be generated by LTGMs will be considered as a set of templates of all existing floor plans that we can invent, although it is not true.

A total of 6 visual artists left worrisome comments that learning how to use LTGMs can become a burden. In the study, LTGMs often generated images somewhat differently from the visual artists’ intention, so they could hardly build the mental model of LTGMs in a short time. After a few trials of LTGMs, P11 (spatial designer) pointed out with a disappointed tone that some visual artists would prefer to draw by themselves or use familiar softwares. Moreover, as suggested by previous work, it is often hard to find a proper text prompt because of the open-ended nature (Liu and Chilton, 2022). For example, P9 (Jewelry designer) and P15 (art instructor) did not know what to type in and contemplated for several minutes to write down the text prompt.

A total of 10 visual artists regarded LTGMs as inefficient in the situations where they already had a specific goal and requirements in their mind. P18 (fashion designer) said if she wants to make pants with several design specifications, using LTGMs would be an inefficient use of time, as it needs a lot of additional time to describe them in sentences, although it cannot guarantee a satisfactory output that exists vividly in her mind. Similarly, P4 (motion graphic designer) said it is often difficult to come up with a precise word (e.g., Baroque architecture) for what she wants, although she has the image (e.g., architectural style lavishly decorated with ornament) in her mind. Last but not least, P28 (animator) mentioned that it is sometimes close to impossible to describe the imagery in words. He said it is important to reflect what cannot be seen in the visual. P28 stated:

P28, Animator: Imagine that I make a girl who is ten years old and grown up as an orphan. Then we may come up with a certain mood of the girl. She might wear a shirt with a low chroma and always carry a cuddly toy around her, which are the things we can infer from her background information… As you know, these kinds of characters are created and developed over a hundred lines of stories. I can hardly think of making a new character with just a few sentences.