GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication

Task-oriented communication between humans and robots is becoming increasingly important, especially in the context of end-user robot programming. Natural language programming allows non-expert users to specify robot tasks through verbal instructions (Connell, 2019; Matuszek et al., 2013; Thomason et al., 2019; Ionescu and Schlund, 2021; Lauria et al., 2002). This approach enables users to communicate complex task requirements and specify reusable programs through multi-turn dialogues (Gorostiza and Salichs, 2011; Stenmark and Nugues, 2013; Thomason et al., 2019), making it a fundamental aspect of end-user robot programming (Ajaykumar et al., 2022; Lieberman et al., 2006).

The integration of artificial intelligence with natural language programming is considered a key method for future household robots and intelligent agents to provide personalized services (Fischer, 2023). Through natural dialogue, this approach enables untrained users without programming knowledge to define reusable robot programs that align with their practical needs. Recent advancements in large language models (LLMs) (Wei et al., 2022) have further accelerated developments in this field. Several studies have explored how LLM capabilities can support end-users in defining robot tasks using natural language (Fang et al., 2024; Gargioni and Fogli, 2024). However, specifying robot tasks based on natural language descriptions of desired program outcomes still poses challenges due to the abstraction gap between natural language and program code (Liu et al., 2023b). Recent advancements in LLM-based end-user programming have investigated structured and visualized “intermediate-level” representations to address the abstraction gap (Liu et al., 2023b; Ge et al., 2024).

LLM-based end-user development (EUD) transforms programming into a collaborative and iterative communication process (Karli et al., 2024; Fischer, 2023). Effective communication between users and robots often requires a continuous cycle of “intention expression → result feedback → intention adjustment” to complete task specification (Glassman, 2023). This process typically integrates multiple modalities to provide user feedback and represent program or task context (Huang et al., 2020; Zhang et al., 2021; Fang et al., 2024), or supports users in expressing intentions through multimodal interactions (Porfirio et al., 2023). Furthermore, human-like multimodal interactions have been explored to facilitate task communication between users and robots (Higger et al., 2023; Huang et al., 2024).

In light of these advancements and the challenges inherent in LLM-based EUD systems, there is an opportunity to enhance verbal programming. This motivates us to draw inspiration from human-to-human verbal communication and explore natural, intuitive interactions that leverage large language models to improve the usability of verbal programming.

Screens on robots serve an important function in enhancing communication by displaying facial expressions and complementing voice interactions (Glauser et al., 2023; You et al., 2020; Chen et al., 2020). Beyond expressive functions, screens also facilitate the communication of complex messages, yet their integration with verbal communication remains an area requiring further exploration.

Currently, the way touch screens are used in service robots has led to their perception as mere “screen bearers”, diminishing the sense of rich interaction with an autonomous agent. In most cases, they are employed as a means to circumvent the current limitations of full speech interaction while also providing an effective way to enable complex interactions (Bonarini, 2020). However, balancing this with other interaction modalities is important to maintain a rich interaction experience. The consistency between voice interactions and graphical user interfaces is crucial for improving system usability and user satisfaction (Okamoto et al., 2009), making it easier for users to understand and operate the system. Research indicates that providing graphical feedback during dialogues significantly enhances user comprehension of the robot’s responses and intentions (Peng et al., 2020).

Visual media can support task communication through various forms, including robot-mounted screens, external display devices, and extended reality (XR) interfaces (Carriero et al., 2023; Maccio et al., 2022; Carriero et al., 2023). Additionally, external objects and gestures can serve as references to assist in task communication, enhancing naturalness and comprehension (Huang et al., 2024; Higger et al., 2023). Some researchers have explored projection techniques that allow robots to visualize task information on physical objects in the environment (Andersen et al., 2016). Compared to single-modality interactions, interactions with robots that exhibit natural multimodal expression provide a more engaging and intuitive communication experience (Breazeal et al., 2016). Drawing inspiration from human-to-human interaction patterns has proven effective in refining robot communication strategies (Huang et al., 2024). Since human-AI agent communication is inherently dialogic, requiring iterative exchanges to refine intent expression, the selection of interaction modalities should be contextually adapted to different scenarios (Glassman, 2023).

Recent work has demonstrated that large language models can serve as intermediaries for multimodal interfaces, enabling both understanding and generation beyond purely natural language content. For example, in graphical user interfaces, LLMs can interpret the semantics and structure of UIs (You et al., 2024; Duan et al., 2023) and generate UI designs (Lu et al., 2023; Kargaran et al., 2023), demonstrating their potential in processing and generating non-linguistic visual representations.

Furthermore, the world knowledge and in-context learning capabilities of LLMs (Zhao et al., 2024) enable the dynamic generation of multimodal interactions tailored to the context on the fly. For instance, GenEM (Mahadevan et al., 2024) utilizes LLMs to flexibly generate and adapt robot expressive behaviors based on natural language instructions and user preferences. Similarly, SiSCo (Sonawani et al., 2024) showcases the ability of LLMs to synthesize both natural language and visual signals adaptively for efficient collaboration. In the context of end-user programming, Cocobo (Ge et al., 2024) illustrates how LLMs can dynamically translate between natural language and visual programming representations.

Unlike traditional rule-based or template-based approaches, these LLM-based methods show the potential for adaptive contextual understanding and immediate generation across multiple modalities. This motivates our exploration of leveraging LLMs to dynamically generate visual aids for task-oriented communication between humans and robots.

We recruited 8 participants (5 female). Each participant was paired with one of the two researchers, forming a total of 8 groups. The recruits consisted of students and teachers from the university. The experiment for each group lasted between 10 and 15 minutes.

Each participant was assigned to a group where they took on the role of maintenance staff (Role B), while the researcher acted as the manager (Role A). Participants were informed of the specific experimental procedures and provided with printed task maps and colored pens.

The scenario was set in a designated college space within the school, where the manager (Role A) verbally assigns tasks to the maintenance staff (Role B). The maintenance staff uses a paper map and a pen to communicate with the manager, ensuring they accurately understand the manager’s intentions and provide a specific execution plan. Two tasks were designed: an exhibition reception task and a patrol task. The detailed scripts can be found in the appendix A.

Before the experiment began, Role A reviewed a script and noted down the tasks they needed to assign to Role B. During the actual communication, Role A was not permitted to refer to the script, and their objective was to ensure the tasks were accurately conveyed to Role B. Role B was responsible for confirming their understanding through questioning and paraphrasing, as well as formulating a concrete execution plan. To facilitate task confirmation, Role B was provided with five different colored pens to annotate pre-printed maps, with no restrictions on paper usage. Each experimental session was recorded on video, capturing participants as they completed the two tasks. The experiment concluded once Role A confirmed that Role B had accurately understood the assigned tasks. Examples of participants’ annotations on the paper maps can be seen in Figure 2. After the experiment, participants were asked about the specific situations in which they used pen and paper, how these tools assisted in articulating their intentions, and what specific benefits they provided.

We analyzed the experiment recordings and synthesized the interview findings, identifying the specific needs and contexts in which users employed visual aids. Based on the experimental data, all participants utilized visual aids by marking or drawing on the map (8/8). Additionally, the majority of participants (7/8) reported that visual aids were beneficial in articulating their intentions.

Based on our formative study observations, we categorized the visual elements used in task communication into four main types: sequence, logic, annotation, and global elements. For each type, we identified specific visual elements (such as lines with arrows, circle annotations, text labels, symbols, colors, and notes) and their representational purposes. For example, sequence-related elements were used to show movement paths and task orders, while logic elements helped represent conditional branches. Annotation elements served to mark locations and add contextual information, and global elements like color coding helped highlight and organize information across different aspects of the task. The complete categorization of these visual elements and their specific uses is shown in Table 1.

Drawing from how humans naturally employ visual aids in communication, particularly their patterns of using visual elements to clarify complex tasks, we formalize the following design considerations for our system:

[DC1]:

Provide continuous and progressive visual feedback throughout the communication process to support step-by-step task understanding and verification, ultimately facilitating the successful completion of task communication.

[DC2]:

Facilitate memory and task comprehension by interpreting user task intentions to plan and organize feedback, including the use of visual aids and speech output, ensuring users stay aware of complex task sequences and spatial relationships.

[DC3]:

Enable effective robot-to-human communication by organically integrating visual aids and speech, allowing the system to convey information more comprehensively and intuitively.

[DC4]:

Leverage visual elements commonly used in human communication (e.g., arrows, labels, symbols) and their associated usage patterns to represent tasks, ensuring clarity and natural alignment with user expectations.

Here we present a sample usage scenario in an office setting to demonstrate how GenComUI and its generative visual aids support multi-turn verbal task communication between humans and robots. The main visual interface of GenComUI is shown in Figure 3.

Lily, a secretary at a tech company, is responsible for coordinating meetings and managing schedules. While she wants to leverage a robot assistant for her daily tasks, she faces two challenges: the robot’s predefined functions are too rigid for her dynamic needs, and she lacks programming expertise to customize robot behaviors. GenComUI addresses these challenges by enabling natural dialogue-based task specification.

In this scenario, Lily wants to create a “visitor reception” task where the robot notifies and guides participants to meetings. After launching GenComUI, she initiates the dialogue through the Voice Interaction Module by tapping the voice button. She says: ”Hello Temi, I would like to develop a visitor reception service.”

The User Intention Understanding Module processes her input and generates an appropriate response: ”Okay, the robot will lead the visitor to the reception area first, then go to the work display area. Do you have any other requirements?” Simultaneously, the Generative Visual Aids Module visualizes the task flow using connected lines and fade-in animations to highlight the newly added requirements (Figure 3-A1) (DC2).

Lily further instructs the robot to then lead visitors to the staff office area and creative studio. The robot acknowledges this and updates the visual aids accordingly (Figure 3-A2).

When Lily modifies the task by saying, ”I want to modify it. After the staff area, lead them to the drawing room,” the robot understands the user’s intent to modify the task steps. The system updates the visualization using fade-out animations for removed elements and fade-in for new ones (Figure 3-A3).

After several rounds of communication, aided by step-by-step visual feedback (DC1), Lily indicates she has finished specifying the task. The robot enters confirmation mode (Figure 3-B), generating synchronized visual and voice presentations of the complete task steps (DC3): ”Step one, activate the service with the keyword ’visitor reception’, then the robot will lead the visitor to the reception area,” ”Step two, robots lead visitors to the exhibition area,” and so on.

Upon Lily’s confirmation, the Task Program Synthesis and Deployment Module generates and deploys the program code. The screen displays the complete task flow with icons, text, and connecting arrows (DC4). After deployment, Lily tests the task through the activated ”Test” button (Figure 3-C). During testing, she identifies the need for the robot to return to the reception area after completing the task. She easily makes this modification by exiting test mode and communicating the additional requirement.

Through this process, Lily successfully creates a customized robot task that matches her specific needs, demonstrating how GenComUI enables non-programmers to naturally specify and refine robot behaviors through multi-modal interaction in an on-the-fly style(Stegner et al., 2024).

The Intention Understanding Module aims to obtain clear task specifications through multi-turn interactions by interpreting user intent, understanding requirements, and tracking interaction progress.

To achieve this, the module leverages the CommunicationGPT model with chain-of-thought reasoning and few-shot learning techniques (detailed in Appendix B.1). The model takes system prompts and dialogue history as input and produces structured outputs containing:

• •

  Task: Current robot task step descriptions based on the ongoing communication

• •

  State: Communication progress tracking to guide the interaction flow

• •

  Speak: Robot’s verbal response content

• •

  Draw: Visual aids type for rendering

These structured outputs are then processed by the backend to generate appropriate robot behaviors, including verbal responses, task-related code updates, and visual aid presentations. See Figure 4.B for the detailed workflow, and Figure 3 for the application of visual aids in the communication process.

The Generative Visual Aids Module dynamically generates graphical interface content during interactions based on the context of task communication using RobotDrawGPT (see Appendix B.2). Based on the structured outputs from the User Intention Understanding Module, the system supports three presentation modes for visual-aided robot-to-human communication:

The Feedback mode highlights modified parts of the task flow, using fade-in animations to emphasize newly added steps and fade-out animations for deleted steps. In Confirm mode, the system synchronizes task steps with corresponding graphical animations and voice descriptions, highlighting each visual element as its associated step is narrated (Figure 3-B1). The None mode simply displays the current task flow without any animations.

The visual content is composed of two main types of elements. Location markers include icons representing robot actions, task numbers, and configurable location colors. These are connected by arrows that feature customizable colors and styles (solid or dashed lines), along with task descriptions. The arrows can establish various types of connections: between two specific locations, from one location to any location, or from any location to a specific destination.

To maintain visual continuity, each rendering process updates only the elements related to the user’s refined intent while keeping other parts unchanged. As shown in Figure 4, during the requirement communication process, the system generates different drawing behaviors according to the user’s intent. The detailed styles and configurable elements are shown in Figure 5, and for implementation details, please refer to RobotDrawGPT’s prompt in Appendix B.2.

GenComUI creates and modifies robot task programs using an LLM based on the task flow input, system prompt (see Appendix B.3), and, if previously generated, existing code and task flow to inform the generation process.

After final user confirmation, the Task Program Synthesis and Deployment Module leverages CodeGenerationGPT (see Appendix B.3) to generate executable robot code based on the confirmed task specifications. The module synthesizes JavaScript programs that orchestrate the robot’s basic commands (Table 2) according to the specified task flow.

Once code generation is complete, the system deploys the program to the robot’s runtime environment and activates the test functionality. Users can then enter test mode to validate the program’s behavior using the specified wake word. The testing interface provides an exit option that returns users to the customization interface for iterative refinement if needed.

GenComUI uses the open-source program temi-woz-android³³3https://github.com/tongji-cdi/temi-woz-android to establish communication between the robot and the backend server via WebSocket⁴⁴4https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API. This enables the backend server to control Temi’s behavior by sending WebSocket commands and receiving callback messages. The backend, developed using Node.js, handles interaction events, manages robot behavior, and makes calls to the OpenAI API. The robot task programs synthesized by the system are written in JavaScript and executed on the backend server. The system uses GPT-4o-2024-08-06 through the OpenAI API⁵⁵5https://api.openai.com/v1/chat/completions, which supports structured output⁶⁶6https://openai.com/index/introducing-structured-outputs-in-the-api/ and provides response times and reasoning capabilities that meet the requirements of this project. We set the temperature parameter to 0 in all API calls. The frontend, built with the React framework and p5.js⁷⁷7https://p5js.org/, dynamically inserts and renders the visual aids code on Temi’s Screen.

To gain deep insights (Greenberg and Buxton, 2008) into how visual aids influence task-oriented human-robot communication, we designed a baseline system that retained all functionalities of GenComUI but omitted the generative visual aids module. While the baseline system displayed only static robot expressions on screen, we provided participants with a paper map identical to the one shown in GenComUI’s interface for reference. This controlled design ensured that both systems had comparable response times and behaviors, allowing us to specifically examine the role of visual aids in human-robot task-oriented communication. By controlling this single variable, we could focus on understanding how visual aids shape the communication process and user experience during task-oriented interactions.

This study recruited 20 participants (12 females, 8 males, aged 18–60, M = 26.5, SD = 8.38) via an online questionnaire, ensuring a diverse range of backgrounds in robot interaction, LLM familiarity, and programming expertise. Half of the participants (10/20) had no prior experience with robots, while the others had experience with home service (8/20) or industrial robots (2/20). Most participants frequently used LLMs (13/20), while some were occasional users (3/20) and others had never used LLMs (4/20). In terms of programming expertise, participants ranged from professionals (3/20) to those with basic skills (7/20) and no experience (10/20).

The experiment was conducted in a simulated office environment as shown in Figure 6. The setup consisted of a Temi robot, a computer running the backend system, video and audio recording equipment for data collection, and printed task guidelines for participants. For the baseline condition, participants were provided with a physical map identical to the one displayed in GenComUI’s interface. Two researchers were present throughout each session: one facilitating the experiment and another conducting behavioral observations.

We designed four spatial programming tasks that required participants to create robot programs involving navigation and actions in an office environment. The tasks were divided into two complexity levels based on the minimum number of required robot commands, ensuring that tasks of the same complexity were interchangeable.

Low-complexity Tasks (minimum 4 robot commands each):

• •

  Task L1: Office Patrolling and Employee Notification
  Program the robot to patrol specific office areas and notify designated employees

• •

  Task L2: Employee Guidance and Area Introduction
  Program the robot to guide an employee through office areas while providing area descriptions

High-complexity Tasks (minimum 8 robot commands each):

• •

  Task H1: Visitor Guidance at Reception
  Program the robot to receive visitors and guide them through multiple office locations

• •

  Task H2: Employee Gathering and Preparation Work
  Program the robot to locate multiple employees and coordinate a meeting preparation sequence

Each participant completed two tasks with each system: one low-complexity task and one high-complexity task. To reduce task-specific biases and improve the generalizability of our findings, the two tasks of the same complexity level were designed to be interchangeable. Task assignment was randomized through a lottery system by the researcher, with the remaining two tasks assigned to the other system condition. This approach ensured a balanced design while examining system performance across diverse cases. See Appendix C.1 for detailed task descriptions.

The experiment followed a within-subjects design where participants experienced both systems. The researcher used a computerized randomization program to determine each participant’s system order, which ultimately achieved a balanced distribution (10:10) across conditions. The procedure for each system condition consisted of three phases:

Training Phase (15 minutes): Participants watched a system-specific tutorial video and received hands-on guidance from researchers on how to operate the system. For the baseline system, participants were provided with a paper map; for GenComUI, the map was displayed on the robot’s screen.

Task Execution Phase (30 minutes): Participants completed two tasks with each system: one low-complexity task and one high-complexity task. For each task, participants communicated their programming intentions through voice commands until the robot indicated understanding by requesting a task trigger word. Upon execution, participants could interrupt and modify the program if needed, with the task concluding only when participants perceived that the robot had successfully executed the intended program.

Assessment Phase: After completing tasks with each system, participants filled out questionnaires.

Following the completion of both conditions, participants engaged in a 20-minute semi-structured interview.

We drew inspiration from human-to-human communication, which often uses visual aids to enhance comprehension, retention, and engagement (Jurin et al., 2010; Davis, 2005). Our study highlighted the importance of continuous visual feedback and the synchronization between verbal and visual elements. Based on these insights, GenComUI was designed to synchronize generative visual interfaces with voice outputs, providing task animations and immediate feedback in line with the dialogue context. This approach offers a customizable visual language presentation that supports both human-human and human-computer interactions (Robbins et al., 1996).

As noted by Glassman (Glassman, 2023), natural language communication between humans and intelligent systems is an iterative loop process, involving command expression, system comprehension, result presentation, and human confirmation, continuously optimizing until true intent understanding and execution are achieved. Our qualitative and quantitative research findings demonstrate that GenComUI’s visual assistance enables users to better convey their intentions, helps them organize their language, and ensures their verbal expressions accurately convey their intentions and are understood by the robot. Additionally, through visual interface feedback, users can more easily gauge the robot’s level of contextual understanding, identify misunderstandings, and make timely modifications or adjustments to information. Users reported better communication experiences with GenComUI than with the baseline system. In complex task communication, we focused on supporting memory retention and ensuring human-robot alignment, areas where GenComUI proved effective in alleviating concerns about forgetting earlier task content. Research results indicated that GenComUI better maintained dialogue under consistent themes, making the communication process more coherent and relevant.

Our results observed that while task completion time showed no significant difference between systems, both Chatbot Usability Scale results and qualitative research reported substantially faster task completion with GenComUI. According to Matthews and Meck (Matthews and Meck, 2016), time perception weakens when attention is drawn to visual information. This suggests that GenComUI’s visual interface engaged users’ attention during communication, and this focused attention facilitated better information reception in human-robot natural language communication (Glassman, 2023), helping users convey their intentions more clearly and ensuring the robot understood and responded as intended (Jurin et al., 2010; Davis, 2005).

From the perspective of LLM-based end-user programming, humans and LLMs collaborate through iterative communication to clarify programming objectives and translation (Karli et al., 2024; Fischer, 2023). This emphasizes the necessity of our lightweight, on-the-fly generation approach (Stegner et al., 2024). Our approach utilizes multimodal representations (Ainsworth, 1999) to integrate generative visual aids into LLM-based end-user development. Rather than using predetermined visual responses in traditional rule-based or template-based visual feedback systems, GenComUI dynamically generates visual aids based on the ongoing communication context and user intentions. This aligns with Fischer’s (Fischer, 2023) vision of adaptive systems.

Natural language programming has long faced an abstraction gap between natural language and program code (Liu et al., 2023b), where natural language often struggles to directly support code generation. This challenge primarily stems from the process where users must organize their language to correctly convey intentions before programming. Our results show that GenComUI effectively assists users in reorganizing their language, particularly in spatial tasks. This spontaneous language restructuring helps users better express their intentions, enhancing the process of natural language-based end-user programming.

Besides, as noted by Fischer (Fischer, 2023), adaptable systems adjust based on user interaction patterns and can tailor themselves to different communication styles. Our results demonstrated that the LLM’s ability to generate personalized visual feedback for diverse task descriptions, while maintaining consistent task outcomes, can further enhance the functionality of visual aids across various scenarios.

Maggioni and Rossignoli (Maggioni and Rossignoli, 2023) proposed that when a robot can engage in verbal interaction, users tend to perceive it as “more human” and are more willing to communicate with it. This was reflected in our study through relatively high perceived intelligence ratings in the Godspeed questionnaire, indicating that both systems demonstrated human-like communication capabilities. However, while some participants reported that visual aids enhanced the human-like nature of the communication, others described the robot as “more like a moving tablet”, experiencing what Bonarini (Bonarini, 2020) termed the “screen bearer” problem. Despite our efforts to mitigate this issue in the current study, fully resolving this challenge remains an open question. Further research is required to develop solutions that enable robot screens to more naturally complement communication while preserving human-like interaction characteristics.

Based on our design objectives and research findings, we summarize design implications for incorporating generative visual aids in task-oriented human-robot communication, focusing on three key themes.