MAxPrototyper: A Multi-Agent Generation System for Interactive User Interface Prototyping

1) UI Composition Knowledge. Rico Dataset (Deka et al., 2017) is one of the most comprehensive open-source UI datasets, including the UI screenshots, their view hierarchy information and their attributes like text, bounds and class, and the composition of these UI elements. We extracted class and bounds from these metadata, and form the UI composition knowledge (¡component types¿¡bounding boxes¿) for each UI.

2) UI Semantic Knowledge. The fine-grained component description can be obtained by parsing content-description from the Rico dataset metadata. We obtain ¡text content/icon descriptions¿.

While Rico contains the metadata of the composition and text contents of UI, it lacks the high-level UI description. We obtain this data through Screen2Words (Wang et al., 2021), which augments Rico to provide  112k high-level textual descriptions for its 22k UI screenshots. Through this dataset, we obtained ¡UI description¿.

Beyond UI functionality semantics, design generation necessitates thoughtful consideration of themes, colors, and the target audience. We adopt Blip2 (Li et al., 2023), which is a zero-shot visual language model, to generate theme descriptions via a visual question-answering approach. We identify four key attributes for theme design: theme color, primary color, theme description, and app category, and create a specific question, pairing it with a UI image, and then input it into Blip2 to derive the answer, which is shown in Table 1. Finally, these three descriptions are concatenated together, and form UI Semantic Knowledge: ¡text content/icon descriptions¿¡high level description¿¡theme design description¿.

We collected the data from Google Material Design Icons (Google, 2023b), a high-quality repository that stores over 900 diverse icons (in SVG format and with text description). We obtain our Icon Knowledge Base: ¡icon SVG code¿¡semantic description¿.

Domain-specific knowledge enhances the accuracy of LLM-generated content. Recognizing the LLM’s token input limitations and the complexities of fine-tuning, we introduce a knowledge retrieval phase, infusing domain-specific knowledge into our generation process.

Based on the user prompt ($In_{p}$) and UI layout ($In_{l}$), we want to retrieve the most relevant knowledge from our large knowledge base. To do so, we concatenate these two information together, and encode them into a vector $\textbf{Emb}(In)$ as the query vector, where $In=In_{p}+In_{l}$. We use the TEXT-EMBEDDING-ADA-002 embedding model (OpenAi, 2023a)). Similarly, we also embed each piece of UI knowledge related to UI Composition and Semantic into a vector ($\textbf{Emb}(kb_{j})$) as well. Then we compute the cosine distance between query and each knowledge, and retrieve the top-k results to instruct our multi-agent system ¹¹1We set $k=2$ in our experiments.. We denote the retrieved knowledge as $refer_{i}$.

Our preliminary experiments, alongside findings by Bryan (Wang et al., 2023a), indicate that when employing related knowledge as few-shot prompting, the initial example tends to be the most influential. Subsequent examples often provide diminishing returns in focusing the model’s output. Furthermore, given the input length limitations of language models, which restrict the number of exemplars in the prompt, we limit the number of references to 2.

Given a user prompt $In_{p}$, a UI layout $In_{l}$, and the top $k$ retrieved knowledge items $\{\sum_{i=0}^{k}refer_{i}\}$ (where, $k=2$), these components are concatenated with the system’s theme description prompt $P_{theme}$ to formulate a comprehensive input for our agent.

Upon completing the theme description generation, the resultant theme description is denoted by $Res_{theme}$.

The objective of generating a theme image is to visually guide the overall design, allowing the sub-agents to generate coherent and consistent design.

We consider Stable Diffusion model (Rombach et al., 2022a), a state-of-the-art text-to-image generation as our main model. However, as this model only considers the text condition, it suffers from limited control over the spatial composition of the image, a crucial aspect for our UI design generation. To address this, we integrate ControlNet (Zhang and Agrawala, 2023), which augments the diffusion model by providing enhanced spatial control over each module. ControlNet controls the generation by manipulating the denoising module by importing additional spatial condition in UNet. We employ UI layout as the spatial condition.

In addition, as the stable diffusion model faces obstacles when generating UI images, which requires a different domain knowledge from general images (Chen et al., 2020b). We further finetune the model using the datasets of UI screenshots and their complementary high-level descriptions collected in Section 2.1.

During sub-agent execution phase, Theme Design Agent identifies the optimal sub-agent corresponding to the component type. We considered 13 component types, as detailed by the Rico dataset. To elaborate, $A_{text}$ handles “Text Button” and “Text”. $A_{img}$ is entrusted with “Image” and “Background Image”, and the $A_{icon}$ focuses on “Icon” components. For other component types, we seek to render them editable, drawing insights from RaWi (Kolthoff et al., 2023b). The color for any component is determined by identifying the dominant RGB color from the image region’s histogram and then representing it in HTML code.

The dynamic between the central agent and the sub-agents is essential to our system’s functionality. When a central agent engages a sub-agent, it uses a combination of past results and a cache pool to inform the sub-agent’s prompts:

In this context, $Res_{t-1}$ is the output from the sub-agent for the $t-1^{th}$ component. The $Cache_{t}$ represents a cache pool that integrates the previous result with accumulated knowledge from earlier iterations. This cache pool serves as an essential memory function, retaining the design context and facilitating multi-turn interactions. Meanwhile, $p_{t+1}$ functions as the prompt for the $t-1^{th}$ component’s sub-agent interaction, incorporating both the specific prompt $p_{sub}$ for the current sub-agent and the cumulative knowledge in $Cache_{t}$.

Baselines. For RQ1, we consider two state-of-the-art image generation models: stable-diffusion-1-5 (AI, 2022) and stable-diffusion-2-1 (AI, 2023) as these two baselines do not incorporate ControlNet module, we consider variants of both by integrating ControlNet, denoted as stable-diffusion-1-5 (with ControlNet), stable-diffusion-2-1 (with ControlNet). In addition, we also employ an ablated version of our approach, MAxProtytper (w/o ControlNet) as a baseline.

Results. As seen in Table 2, For generation quality, we consistently outperforms both baseline models in terms of FID scores, regardless of whether ControlNet is utilized. Specifically, a lower FID score suggests that the distribution of generated images more closely matches that of real images. This indicates that MAxPrototyper’s outputs are more realistic, evident from its significantly reduced FID scores: 33.08 without ControlNet and an even lower 23.76 with ControlNet.

For diversity, Notably, GD scores reflect the model’s ability to produce varied yet detailed UI designs. A higher GD indicates more detail, suggesting that the designs are diverse and intricate in their presentation. MAxPrototyper’s GD scores, both with and without ControlNet, surpass those of the baseline models. This increase in GD values emphasizes our model’s superior capability in producing designs that are varied and enriched with details compared to its peers.

For ControlNet, Incorporating ControlNet results in noticeable improvements in FID scores for all models. For our MAxPrototyper, the FID shows an enhancement of 10.68, representing a substantial 32% improvement.

In conclusion, Our MAxPrototyper outperforms the baseline models in terms of both quality and diversity. The addition of ControlNet optimizes this performance further by introducing layout constraints, reinforcing its potential for generating realistic and detail-oriented UI designs.

Baselines and Ablation Strategy. To better comprehend the interaction and individual impact of MAxPrototyper’s components on the end result, we perform an ablation study, sequentially removing each module and evaluating the effect. As seen in Table 3, we carefully crafted six ablations.

Results. MAxPrototyper’s analysis highlights each module’s critical role in achieving high-quality, diverse UI designs. The “Knowledge Retrieval” module is essential for data importation, with its removal increasing the FID score from 23.76 to 42.56, significantly impacting design quality. “Theme Image Generation” is vital for linking text and visuals, sees the FID score rise to 33.08 when omitted, underscoring its significance in visual integration. The “Sub-agent Execution” phase, with specialized agents for each design element, demonstrates the value of a detailed generation approach through minor FID score variations.

In conclusion, “Knowledge Retrieval” and “Theme Image Generation” are the most influential, but all modules collaboratively enhance the final output.