Artworks Reimagined: Exploring Human-AI Co-Creation through Body Prompting

Public displays offer opportunities to study how users interact with technology. However, public displays bring unique challenges, such as display blindness (Müller et al., 2009), interaction blindness (Hosio et al., 2016), and the first-click problem (Kukka et al., 2013). The latter arises when users are unsure how to initiate interaction with a public display. In this section, we focus on challenges related to the public setting of our study.

Privacy concerns require consideration. Brudy et al. identified ‘shoulder surfing’ – where unauthorized individuals may view sensitive information – as a pertinent risk associated with public displays (Brudy et al., 2014). Further exploring privacy issues, Memarovic examined how the public’s concerns about photography in the vicinity of public displays influence user interaction (Memarovic, 2015). Their study emphasized the importance of transparent policies regarding the storage and usage of photos taken in these contexts, highlighting a general need for clear communication about the handling of users’ data. In a comparative study, Collier et al. addressed differences between private and public self-service technologies (Collier et al., 2014). Their findings suggest that perceptions of control and convenience significantly vary depending on the setting, with public technologies often viewed as less controllable and convenient than their private counterparts (Collier et al., 2014). This perception could influence user engagement and satisfaction with the public display in our study. Hosio et al. studied privacy in interactive public displays (Hosio et al., 2019; Alorwu et al., 2022). They found that people are surprisingly willing to volunteer highly sensitive personal information in public settings.

These studies reveal a complex landscape of user behavior and expectations when interacting with displays in public spaces. Addressing these concerns is crucial for designing an interactive public installation.

Embodied interaction can provide an engaging experience. This is particularly relevant in the context public cultural spaces, such as GLAM institutions. In museums, the contact with more traditional art forms, namely painting or sculpture, tends to occur passively through a cognitive engagement with the artworks. Digitally enriched museum experiences (Calvi and Vermeeren, 2023) open new opportunities to experience and interact with such artworks through embodied cognitive engagement via human-technology interaction. Gestures and posing are an engaging way of meaning making in the GLAM context (Steier et al., 2015; Steier, 2014; Fu et al., 2024). In the remainder of this section, we review prior work on embodied interaction in the context of GLAM institutions and other public settings.

Trajkova et al. explored engaging museum visitors with embodied interaction (Trajkova et al., 2020). Their study was aimed at human-data interaction, enabling people to explore data sets and interactive visualizations with gestures and body movements.

Oppenlaender and Hosio presented a system for museum visitors to provide feedback for artworks, including selfie photos and selfie videos (Oppenlaender and Hosio, 2019). The latter was perceived as a surprising way of interacting with an artwork and its artist. Kozinets et al. also explored museum selfies (Kozinets et al., 2017), finding selfies to be a way to construct narratives about oneself. However, their study did not feature embodied interaction with dynamically generated artifacts in the museum context.

Positive effects of interactive technological engagement with artworks have been reported. For example, aesthetic appreciation (including interest, intensity, pleasure, and learning) was higher in artworks with interactive elements compared to non-interactive physical objects (Jonauskaite et al., 2024). In Coeckelbergh’s framework, body prompting can be conceptualized as a performance-based approach to interaction and AI image generation (Coeckelbergh, 2023). Body prompting is a poietic performance within human-technology co-creation. The performance-based approach emphasizes the process of interaction in which novel artistic (quasi-)subjects and objects emerge and are produced in and by the processes instead of instruments and tools (Coeckelbergh, 2023).

Art museums are increasingly digitizing their collections and making them accessible on the internet thus fostering different reinterpretations and re-creations of the paintings (Barranha, 2018). Several experiments have shown that people enjoy re-creating and reinterpreting famous artworks. Launched at the height of the COVID-19 pandemic, The Getty Museum Challenge was a widely successful initiative, which invited the public to re-create an artwork with household items (Romero, 2020). In 2023, the Mauritshuis Museum in The Hague launched another challenge, which invited the public to reinterpret Vermeer’s “The Girl with a Pearl Earring” using different processes, including generative AI (Mauritshuis Museum, 2023). The AI-generated images were later exhibited in the Museum in place of Vermeer’s famous painting, in a digital loop display which featured a wide range of experiments produced with different media. In parallel to such institutional initiatives, generative AI has also been used by researchers to reinterpret artists’ self-portraits (Barranha, 2023).

These examples of re-creating and reinterpreting artworks suggest that new activities designed to explore art collections with the use of generative AI can foster participatory creative experiences. However, the Getty and Mauritshuis Museum challenges were completed from the participants’ homes, re-creating or reinterpreting the artwork either with body poses and household items or generative AI. In our work, we explored how visitors could be engaged to re-create or reinterpret an artwork with generative AI and their bodies at a public event.

Generative AI has been explored in the context of public exhibitions. In this section, we review three interactive installations that used generative AI.

The artist duo Varvara & Mar presented an art installation in which the human impact on a landscape is gradually revealed using eye-tracking and generative AI (Canet Sola and Guljajeva, 2024). While their interactive art installation requires bodily presence, it is driven by gaze-based interaction with the public display, not body posing.

Benjamin and Lindley explored embodiment in Shadowplay (Benjamin and Lindley, 2024). Users of this AI art installation produced shadows on a wall, which served as input for the image generation AI. Our system differs in that the users’ body directly serves as input to re-create or reinterpret an existing artwork via body prompting.

GenFrame by Kun et al. (Kun et al., 2024a, b) is a more recent generative art piece designed for the museum context. GenFrame’s picture display gives the appearance of a traditional artwork, however the audience can control the image generation with three rotating knobs. Like our work, GenFrame was implemented with ControlNet and StableDiffusion. Nevertheless, the interaction affordances of GenFrame are limited by design. Our work explores body prompting as an expressive way of providing input to the image generation process.

The European Researchers’ Night is an annual event, which aims to interest the general public in research. It is one of the biggest science events in Finland, held at several universities nation-wide. The event gives the public the opportunity to discover the wonders of science in fun and inspiring ways. As a multidisciplinary event, Researchers’ Night offers a wide range of activities from workshops to laboratories, presentations, and lectures. The event program caters to the general public, including visiting school groups and parents with children.

Our study was born from our previous experience hosting an exhibit at the European Researchers’ Night in 2022 (Oppenlaender et al., 2023c, b). At the time, text-to-image generation was still a new phenomenon, and many visitors had not tried it before. The exhibition stand consisted of two laptops on which visitors could try out three text-to-image generation systems (Midjourney, Stable Diffusion, and DALL-E). One laptop was connected to a large public display. During this event, we observed that while interesting for the person writing the prompts, textual interaction with the image generation system was not very appealing for bystanders in a public audience, due to prompts being written in private. In the following year, we designed an interactive art installation to address this issue.

The design of our installation had three goals. First, we sought to overcome the limitations in engagement that we had observed in 2022 and provide a highly engaging experience for visitors of the local Researchers’ Night event. Second, we aimed to allow visitors to reconfigure an existing artwork with an input modality other than typing. To this end, we selected posing (“body prompting”) as input modality. Visitors would be able to reimagine an existing artwork by posing in front of a camera. Third, like a camera booth, the installation was designed to be easy to use and self-guiding, allowing its user to go through the steps necessary to take a selfie photo.

We created an interactive art installation, called Artworks Reimagined, that guides its users through four simple steps (depicted in Figure 1 and described in detail in Section 3.3). The physical setup of the installation is depicted in Figure 2. Divider walls were used to define four separate areas. The public posing stage allowed participants to select an artwork on the touch screen and pose in front of the camera. This setup was mirrored in the private camera booth which was cordoned off and protected from viewing with divider walls. Results from the public booth were viewed on a large screen controlled by a laptop, whereas results from the private booth were viewed in private on two laptops equipped with privacy screens.

We selected artwork images for the installation from two sources. The first source included paintings from the “Golden Age of Finnish Art” (circa 1880-1910). We identified artworks by visiting Wikipedia pages of prominent artists from that time period.²²2https://en.wikipedia.org/wiki/Golden_Age_of_Finnish_Art The second source was a list of popular paintings on WikiArt.³³3https://www.wikiart.org/en/popular-paintings/alltime When selecting artworks, we avoided nudity and other depictions not appropriate for public viewing in the presence of children. This convenience sample of 60 artworks (29 from the Finnish Golden Age and 31 from WikiArt) represents a mix of well-known international masterpieces and artworks with a deep local context.

The aim of our study was to observe and understand how body prompting is used and experienced in the context of a public event. Therefore, the research was conducted “in-the-wild”, focusing on observing participants in an environment outside of the controlled laboratory, without interference from researchers. The research was exploratory and participants were not given specific posing instructions.

The study procedure is summarized in Figure 1. Participants were invited to walk up to the public display. The first application screen consisted of a consent form informing the participants about the extent and use of the collected data. Participants were specifically informed that only an anonymous, stick figure-like pose would be collected, not the original photo. After providing consent by clicking a button, participants selected an artwork from the list by touching the screen. Next, participants were informed that a timer would count down from 10 seconds. After confirming this message, participants could see themselves on the screen and assume a pose while the timer counted down to zero. The posing was supported by showing the artwork for a brief moment before displaying the camera feed, and a small version of the artwork was additionally shown in the corner of the screen while posing. Participants were not given any instructions on what specific pose to assume and were completely free to deviate from the source artwork. Finally, participants were shown a short unique code from a curated list of Finnish and English words, generated with a language model, and informed to exit the booth in the final application screen. The short code was used to associate the subsequent interview data with the generated image.

Following the interaction with the installation, one of five research assistants invited the participants to a semi-structured interview recorded using a mobile phone. The assistants showed the participant(s) the artwork and detected body prompt during the interview and recorded participant(s)’ reactions and comments in response to seeing the generated artwork.

The study was conducted in accordance with the policies of the University of Jyväskylä’s Institutional Review Board (IRB) and the Finnish National Board on Research Integrity (TENK). Children under the age of 18 were allowed to use the installation under the supervision of adults, but were excluded from the interviews. In this article, we only focus on the data collected from 79 adults who agreed to an interview.

We developed an interview guide consisting of 12 questions. Besides basic demographics, the research assistants asked participants about their motivation for selecting the artwork, the body prompt, and the choice of private or public posing area. The interviewers also inquired how participants experienced body prompting and what they wanted to express with their body prompt. The structured questions were followed by open-ended follow-up questions (“can you describe why?”). A Likert-scale question captured how enjoyable the experience was for the participant (from 1 – Not At All Pleasant to 5 – Extremely Pleasant). The interview also included the ten-item questionnaire on key personality traits (commonly referred to as “Big Five”) (Digman, 1990).

The generated image was revealed on the screen at the end of the interview and participants were asked whether the image expressed their original intent, thus giving insights into co-creative aspects of body prompting, and any other thoughts about their body prompt and the generated image.

The interview data was transcribed using a text-to-speech service, translated from Finnish into English using Google Translate, and manually reviewed and corrected by three research assistants who listened to the recordings and manually corrected translation errors. The qualitative data was analyzed following a grounded theory approach (Glaser and Strauss, 1967). Three authors (one Postdoc in Cognitive Science, one PostDoc in Computer Science, and one PhD student with professional experience in user experience design) discussed a preliminary coding scheme for each question as a starting point for inductive coding. The three authors then individually coded the responses of a sample of 30 participants. In a discussion following this individual coding session, the authors noticed that coding the data was straightforward in all cases (McDonald et al., 2019). That means, in all but a very few cases, it was easy to identify themes in the data, and in two other questionnaire items, the responses were too diverse to identify common themes. Therefore, it was decided that one author would inductively code the interview data.

The generated images and the body prompts (i.e., poses) were rated by the three authors along three criteria: a) dynamism of the source artwork (low/medium/high), b) dynamism of the participant’s pose (low/medium/high), and c) a comparison of the source artwork and generated image in terms of whether a change in narrative took place. For a) and b), the authors collaboratively agreed on a codebook, as listed in Table 2 in Appendix A. With the metric of ‘dynamism’, we intend to capture the expressiveness of the artwork and body prompt. A static pose was rated to be of low expressiveness, whereas a highly dynamic pose was rated as being very expressive. For c), the authors agreed to code a change in narrative if a subject engaged in a different activity or had a different expression (e.g., static versus humorous), or if the model hallucinated artifacts that caused or strongly contributed to a change in narrative. After a first rating round, the ratings were discussed and each author revised their ratings for internal consistency.

Employing Fleiss’ Kappa for inter-rater reliability assessment, our study found high agreement levels among the three raters across all three variables: source artwork dynamism ($\kappa=0.96$), pose dynamism ($\kappa=0.86$), and narrative change ($\kappa=0.88$). These results underscore the raters’ cohesive interpretations and a consistent and robust evaluation process for the studied variables. After calculating inter-rater agreement, a majority vote between the three authors was used to produce the final set of ratings.

In addition, one author analyzed and coded the difference between the source artwork and the body prompt, characterizing the latter as either being an imitation (re-creation) of the source artwork or an attempt to ‘reimagine’ (reinterpret) the artwork with a body prompt that significantly differs from the source artwork. The latter equates to wanting to change the narrative between source artwork and generated artwork via the body prompt. It was further coded whether this narrative change in the resulting image originated from the generative model (e.g., by hallucinating artifacts or facial expressions) or the user (via the body prompt), with options ‘model’, ‘user’, ‘both’, and ‘n/a’ (meaning no major change in narrative took place).

We developed a software architecture based on T2I-Adapter (Mou et al., 2023) with four different inputs (depicted in Figure 3):

1. (1)

   Pose: The pose was detected from the body prompt photo and used to guide the image generation with openpose-based ControlNet (Zhang et al., 2023; Cao et al., 2021).

2. (2)

   Artwork: Style information was extracted from the selected source artwork and used as an input for the T2I-Adapter.

3. (3)

   Prompt: A textual prompt was generated from the selected source artwork using CLIP-Interrogator (version 0.6.0).⁴⁴4https://github.com/pharmapsychotic/clip-interrogator

4. (4)

   Negative prompt: A negative textual prompt was manually designed, based on best practices in text-to-image generation (Oppenlaender, 2023). This negative prompt was used for all image generations to improve the quality of outputs and prevent unsafe generations (Qu et al., 2023).

Images were generated with Stable Diffusion v1.5 (Rombach et al., 2022) with 50 steps and cfg 8.0. The generated images were upscaled twice with Real-ESRGAN for viewing on the large public display. GFPGAN was used to enhance faces. The image generation architecture was packaged as a cog container and uploaded to Replicate.com where it is publicly available for inference via an API.⁵⁵5https://replicate.com/joetm/camerabooth-openpose-style

Two React-based web applications were developed, one for body prompting and one for viewing the results. Both applications were hosted in AWS S3 buckets, using an ssl-encrypted AWS Cloudfront to provide a secure connection.

The first application guided users through the process with a linear application flow (c.f. Figure 1). The application captured the photo from the webcam MediaStream using the MediaDevices API once the timer hit zero. An AWS Gateway was set up to proxy requests from the web application to Replicate’s API. Results from the API were received with a webhook (AWS Lambda function) and stored in a JSON file hosted in an S3 bucket. The application was automatically reset after 60 seconds once the final application screen was reached. The main purpose of this timed reset was to reduce queuing for participants at the public viewing area — the bottleneck of our study design — and to give more time to the subsequent interview. The source artworks were shuffled after each image generation.

The second application enabled the research assistants to scroll through the generated images with keyboard arrow keys. The application used long-polling to fetch the list of generated images. A red square on the screen indicated the number of newly generated images available for viewing. The participants pose was shown in the corner of the screen. Generated images were viewed either on a public display or in private on two laptops with privacy screens, depending on which posing area was chosen by the participant.

Interviews were conducted with 79 participants after they used the installation. Participants were on average 37 years old ($Median=35$, $SD=13.5$, $Max=68$) and included 46 women, 32 men, and one non-binary participant.

Participants’ Big Five personality traits — Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism — were assessed on five-point Likert scales. Participants’ above average openness scores ($Mean=3.56$, $SD=0.83$) suggest a general tendency towards creativity and openness to experience within the cohort. Conscientiousness followed closely ($Mean=3.51$, $SD=0.72$), indicating a high level of reliability and organization among participants. Extraversion and Agreeableness scores were slightly lower ($Means=3.20$ and $3.19$, respectively, $SDs=0.56$ and $0.72$), pointing to moderate levels of sociability and empathy. Neuroticism received the lowest average score ($Mean=2.93$, $SD=0.76$), suggesting a lower degree of emotional instability and stress sensitivity in the group. These findings offer a nuanced view of the personality composition within the participant sample, highlighting a tendency towards openness and conscientiousness, and providing context for our exploration into the behaviors of the study cohort.

Due to the study being conducted in the field without control over the number of participants per condition (i.e., group versus solo participation, body prompting strategy, artwork choice, and public versus private posing area), we focus on describing the images in Section 4.2 and qualitatively analyzing the interview data in sections 4.3–4.6. For a rich description, we contextualize participant quotes with the codes listed in Table 1.

For participating groups, interviews were held with one group member, but other group members were allowed to comment. For simplicity, we refer to both groups and single participants as ‘participant’ in this work. In the following section, we describe the images generated at the event.

In total, 172 images were created during the six-hour event (112 images in public and 60 images in the private booth). Due to the diversity of source artworks, the body prompts are difficult to compare. In this section, we provide a brief descriptive overview of the 172 generated images and body prompts. This section mainly reports on general trends, while specific outstanding instances are highlighted in Section 4.6.1. A gallery of all generated images is available at https://artworksreimagined.com.

Overall, the style of the source artwork was reproduced well, to a degree where it was clearly recognizable which source artwork was used as style reference (see Figure 4). However, the generated images captured the source artwork to varying degrees, leaving room for interpretation and surprise.

The body prompts had varying degrees of expressiveness. Some participants simply stood motionless in front of the camera in a neutral pose with arms dangling beside their hips. Other participants were more dynamic and expressive, raising limbs into the air. A few participants went to great lengths to produce their body prompts (as depicted in Figure 5, bottom center). The generative model’s output would reflect the same number of posing subjects. Overall, there were no noticeable differences between the body prompts performed in public and private, with a few exception (described in Section 4.6.1).

We classified whether a change in narrative had taken place between source artwork and generated image. In almost half of the generated images (46.8%) a change originated from the participant’s body prompt, which means the pose introduced significant modifications in the generated image. In another 29.1% of the generated images, no change in narrative took place and the new generated image appeared to be a reproduction of the source artwork’s original idea, with some minor changes (e.g., number of subjects).

About a quarter of the participants (24.1%) encountered surprising generative artifacts in the generated image. Co-creation with AI can involve surprising hallucinations by the generative model. With hallucinations, we specifically refer to artifacts introduced by the generative model that cannot be found in the source artwork or in the participant’s body prompt, often resulting in a change in narrative between the source artwork and the generated image. In this study, hallucinatory artifacts included door frames, wooden sticks, fields with houses in the distances, animals, and more. However, these generated elements in the image did not necessarily deviate from the narrative portrayed in the source artwork.

About 8% of the participants encountered surprising artifacts originating from the generative model that clearly changed the narrative. Another 16.2% of participants encountered images where the narrative change originated both from the model and the participant. These were often images where the participant’s pose deviated from the source artwork, with the AI adding surprising elements, such as additional characters or changes to the subjects. In other instances, the model added minor hallucinatory elements that still resulted in a change of narrative. For instance, in the generated image in Figure 5, bottom right, the model appears to have added COVID face masks which are not present in the source artwork. In another funny example, the generative model seemingly changed the gender of a subject from a blonde woman to bald man (see Figure 5, middle row, center).

The participants’ key decisions can be summarized as follows:

• •

  Artwork choice: 52 participants (66%) selected an international artwork from Wikiart and 27 (34%) a local Finnish artwork.

• •

  Camera booth choice: 46 participants (58%) used the installation’s public posing stage, compared to 33 participants (42%) who posed in the private booth.

• •

  Participation mode: 44 participants (56%) participated as a group compared to 35 participants (44%) who body prompted individually.

In the following, we describe why participants selected a given source artwork which will provide context for understanding their body prompts.

Our installation was designed to provide a choice of body prompting in public or private environments. Almost half of the participants ($n=35$) had no strong opinion about this choice and did not reveal many concerns about privacy. A significant number of participants ($n=29$) selected to body prompt in public or private for the pragmatic reason of having a shorter wait in line. H11, for instance, selected the public booth “because it was free and the other one was occupied” while group A35 “wanted the others to see what [they were] doing” (A35). Seeing the artwork on the big public screen, as opposed to the smaller laptop screen, also motivated some participants to select the public booth.

A minority ($n=11$, 14%) mentioned they selected the private booth out of shyness and not wanting to pose in public considering that it provided a safe space where they “could be more [themselves] or do crazier things” (D76) and there was not “everyone staring” (G66, B73), which was perceived as embarrassing by a few participants (e.g., D51, F62). The private booth provided an environment to this group of participants with “no rush” (B72) and “not so much pressure” (B71, F53). However, F62 called her preference for the private booth “unnecessary shyness”, as seeing others perform in public was fun and enjoyable to her, but concerns kept creeping up when she was faced with the decision to pose in public herself.

The Big Five personality factors were not found to correlate with the choice of posing area (public vs private), participation mode (group vs alone), source artwork (international vs local), dynamism of the source artwork, and expressiveness of the body prompt (Spearman’s rank correlation, $p>0.05$), with two significant, but weak exceptions. The first correlation was found to be in the personality trait of agreeableness: participants who were more agreeable tended to select a more dynamic source artwork; $\rho=0.237$, $p<0.05$, Cohen $d=0.50$ (CI [0.04, 1.01]). Second, there was a reverse correlation between extraversion and the expressiveness (dynamism) of the body prompt; $\rho=-0.236$, $p<0.05$, Cohen $d=-0.54$ (CI [-1.01, -0.10]). This indicates that for participants with high level of extraversion, the expressiveness of their body prompt tended to be low. The finding may appear counterintuitive since extraversion is typically associated with sociability, enthusiasm, and positive emotions, traits that may be associated with more dynamic and expressive poses. The inverse correlation could suggest that extraverted individuals, perhaps due to their comfort with social interactions, may not feel the need to convey their personality as strongly through a single body prompt, or they might approach body prompting with a different mindset compared to more introverted individuals. However, the correlation was very weak, as the expressiveness of the body prompt is influenced by multiple factors beyond a participant’s level of extraversion, such as the context, the individual’s mood, or the cultural background.

Generative AI has ushered in an era of human-AI co-creation (Kantosalo et al., 2020; Rezwana and Maher, 2023; Capel and Brereton, 2023). The human agency in this co-creative process varies on a spectrum (Parasuraman et al., 2000), from full human autonomy to full machine automation. In our study, participants experienced human-AI co-creation along this spectrum. Depending on the chosen source artwork and body prompt, the generative model had varying levels of influence on the generated image. In combination with the participants’ body prompting strategy, their experiences included either re-creation or reinterpretation, as discussed in Section 2.3. The participants pursued their strategy with varying levels of tenacity, but most interactions were casual, as participants picked a level of interaction suitable for their desired engagement (Pohl, 2015; Pohl and Murray-Smith, 2013).

While it can be argued whether the body-prompted images are art (McCormack et al., 2023; Coeckelbergh, 2023), the poietic act of performing a body prompt changed the dynamic of the creative process of image generation, placing the human in the center of the co-creative process in a public setting. In Kun et al.’s GenFrame, participants turned knobs to control image generation parameters, but otherwise relegated the creative process fully to the machine (Kun et al., 2024a). In our co-creative study, the AI became “more than just a machine”, with AI and humans being co-creators and co-performers (Coeckelbergh, 2023). This highlights the importance of interaction design in co-creative systems (Rezwana and Maher, 2023).

In this section, we reflect on the few shortcomings in the design of our study and installation, and draw on this experience to provide recommendations for practitioners interested in the design of installations and researchers conducting user studies involving interactive image generation in a GLAM context.