LayoutPrompter: Awaken the Design Ability of
Large Language Models

Let’s consider a conditional layout generation task. We denote its training set as $\mathcal{D}=\{(x_{j},y_{j})\}_{j=1}^{M}$. Here, $(x_{j},y_{j})$ represents the $j$-th sample of $\mathcal{D}$, which is an (input constraint, output layout) pair, and $M$ is the total number of samples. As illustrated in Figure 2, for a given test query $x_{\text{test}}$, the in-context learning prompt $\mathbf{P}$ is composed by sequentially concatenating a task-specific preamble $R$, $N$ exemplars and the query itself:

$$\mathbf{P}=[R;F_{X}(x_{k_{1}});F_{Y}(y_{k_{1}});\ldots;F_{X}(x_{k_{N}});F_{Y}(y_{k_{N}});F_{X}(x_{\text{test}})],\quad\{k_{i}\}_{i=1}^{N}=G(x_{\text{test}},\mathcal{D}).$$

(1)

To be more specific, the preamble $R$ provides the essential information about the target task, such as the task description, layout domain and canvas size. $F_{X}(\cdot)$ and $F_{Y}(\cdot)$ are serialization functions that transform task input $x$ and output $y$ into sequences, respectively. $G(\cdot,\cdot)$ denotes an exemplar selection function, which retrieves the in-context exemplars from $\mathcal{D}$ according to $x_{\text{test}}$. The details of $F_{X}$, $F_{Y}$ and $G$ will be elaborated in the following sections.

Notably, when the number of exemplars $N$ is set to $0$, few-shot in-context learning degenerates to zero-shot learning, where LLMs predict the test output $y_{\text{test}}$ solely based on the preamble $R$ and $x_{\text{test}}$. In our experiments (see Section B.2 in Appendix), we find that additional exemplar guidance can help LLMs better comprehend the task and grasp the rough pattern of the required layouts. Hence, we opt for few-shot learning ($N>0$) instead of zero-shot learning ($N=0$) in this work.

To begin, we first establish some notations. For each element $e$ that constitutes a layout, we describe it by its element type $c$, left coordinate $l$, top coordinate $t$, width $w$ and height $h$, i.e., $e=(c,l,t,w,h)$. Here, $c$ is a categorical attribute. The other four are numerical geometric attributes, which will be discretized in the implementation (see Section A in Appendix).

Input Constraint Serialization. For constraint-explicit layout generation, the input constraints are element-wise constraints on $e$. We serialize such constraints in the same way as LayoutFormer++ [15], where they represent each constraint as a sequence and then combine different constraints through concatenation. For example, if $x$ specifies the element types and sizes, $F_{X}(x)$ takes the form of $F_{X}(x)=\texttt{"}c_{1}w_{1}h_{1}|c_{2}w_{2}h_{2}|\ldots\texttt{"}$. In this work, we adopt these ready-made sequences for constraint-explicit layout generation tasks. Regarding content-aware layout generation, the image nature of the input canvas poses a unique challenge for serialization, i.e., enabling LLMs that can only read text to perceive image content. Inspired by DS-GAN [9], we recognize that the saliency map [8] can well capture the key content shape of a canvas while discarding other high-frequency, irrelevant details (see Figure 3). To facilitate serialization, we further convert it into a rectified saliency map $m=(l_{m},t_{m},w_{m},h_{m})$ by detecting region boundaries with pixel values greater than a certain threshold. After preprocessing, the input canvas $x$ can be represented in a format understandable by LLMs: $F_{X}(x)=\texttt{"Content Constraint:}\texttt{ left }l_{m}\texttt{px,top }t_{m}\texttt{px,width }w_{m}\texttt{px,height }h_{m}\texttt{px"}$. For the text-to-layout task, where natural language descriptions are used to generate layouts, the constraint sequence is simply the input text itself.

Output Layout Serialization. For the output $y$, we propose to serialize it into the HTML format that LLMs are more familiar with and good at, rather than the plain sequence used in prior works [15, 7]. Following common HTML representation, we denote the complete output sequence as a concatenation of multiple HTML segments $a$: $F_{Y}(y)=[a_{1};a_{2};\ldots]$. Here, the $i$-th segment $a_{i}$ represents the $i$-th graphic element $e_{i}$ of $y$. It specifies the element attributes in the following format:

$$\texttt{<div class="}c_{i}\texttt{" style="}\texttt{left:}l_{i}\texttt{px; top:}t_{i}\texttt{px; width:}w_{i}\texttt{px; height:}h_{i}\texttt{px"></div>}.$$

(2)

Thanks to the powerful in-context learning ability of LLMs, the test output $y_{\text{test}}$ will be predicted in the same HTML format, making it easy to extract the required element attributes from the output. More input-output examples can be found in Section D of the supplementary material.

As mentioned above, $G$ selects $N$ in-context exemplars that have the most similar layout constraints to $x_{\text{test}}$ from $\mathcal{D}$. The selected exemplars are randomly shuffled and combined to construct $\mathbf{P}$ (see Equation 1), thereby enhancing LLMs’ understanding of various constraints. To achieve this, we design an evaluation suite $s$ to measure the constraint similarity between the test query $x_{\text{test}}$ and each candidate exemplar $(x_{j},y_{j})\in\mathcal{D}$. Then, $G$ can be further expressed as a Top-k selection function:

$$G(x_{\text{test}},\mathcal{D})\triangleq\text{Top-k}(\bigcup\limits_{(x_{j},y_{j})\in\mathcal{D}}\{s(x_{\text{test}},x_{j})\},N).$$

(3)

Since we divide existing layout generation tasks into three categories, each with distinct input constraints, their similarity measures have different representations. We’ll elaborate below.

Constraint-Explicit Layout Generation. As constraint-explicit layout generation tasks only consider element-wise constraints, we define $s(x_{\text{test}},x_{j})$ using inter-element constraint similarities. Specifically, we construct a bipartite graph between $x_{\text{test}}=\{p_{\text{test}}^{u}\}_{u=1}^{U}$ and $x_{j}=\{p_{j}^{v}\}_{v=1}^{V}$, where $p$ denotes the element-wise constraint on $e$. $U,V$ are the constraint numbers of $x_{\text{test}},x_{j}$. Then, the inter-element similarity $W$ (i.e., the weight of bipartite graph) and the overall constraint similarity $s$ are defined as:

$$\vspace{-2mm}s(x_{\text{test}},x_{j})\triangleq\frac{1}{|\mathbb{M}_{\text{max}}|}\sum_{(p_{\text{test}}^{u},p_{j}^{v})\in\mathbb{M}_{\text{max}}}W(p_{\text{test}}^{u},p_{j}^{v}),\quad W(p_{\text{test}}^{u},p_{j}^{v})=\mathds{1}(p_{\text{test}}^{u},p_{j}^{v})2^{-\|\mathbf{g}_{\text{test}}^{u}-\mathbf{g}_{j}^{v}\|_{2}}.$$

(4)

Here, $\mathds{1}$ is a 0-1 function equal to 1 if $p_{\text{test}}^{u}$ and $p_{j}^{v}$ specify the same element type, and 0 otherwise. This ensures that constraint similarity is only considered between elements with the same type. $\mathbf{g}_{\text{test}}^{u}$ and $\mathbf{g}_{j}^{v}$ are specified geometric attributes of $p_{\text{test}}^{u}$ and $p_{j}^{v}$. Given the edge weight $W$ of the bipartite graph, we adopt Hungarian method [20] to obtain the maximum matching $\mathbb{M}_{\text{max}}$. And $s(x_{\text{test}},x_{j})$ is calculated as the average weight of matched edges (as shown in Equation 4).

Content-Aware Layout Generation. The constraint of content-aware layout generation is the input canvas. The similarity of two canvases $x_{\text{test}},x_{j}$ is defined as the IoU (Intersection over Union) of their rectified saliency maps (see Section 3.2) $m_{\text{test}},m_{j}$:

$$s(x_{\text{test}},x_{j})\triangleq\text{IoU}(m_{\text{test}},m_{j})=\frac{|m_{\text{test}}\cap m_{j}|}{|m_{\text{test}}\cup m_{j}|}.$$

(5)

Text-to-Layout. We leverage the CLIP [29] text encoder to encode input texts into embeddings. The constraint similarity $s(x_{\text{test}},x_{j})$ is defined as the cosine similarity of input text embeddings $n_{\text{test}},n_{j}$:

$$s(x_{\text{test}},x_{j})\triangleq\frac{n_{\text{test}}\cdot n_{j}}{\|n_{\text{test}}\|\|n_{j}\|}.$$

(6)

People usually judge the quality of generated layouts from two perspectives: (1) whether they are visually pleasing; (2) whether they look like the real layouts. Therefore, our proposed layout ranker follows the same principles to evaluate layout quality. To be more specific, it measures the quality of an output layout using a combination of metrics:

$$q(y_{\text{test}})=\lambda_{1}\mbox{Alignment}(y_{\text{test}})+\lambda_{2}\mbox{Overlap}(y_{\text{test}})+\lambda_{3}(1-\mbox{mIoU}(y_{\text{test}})).$$

(7)

Here, $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyper-parameters to balance the importance of each metric. Alignment and Overlap reflect quality from the perspective (1), while mIoU mainly focuses on perspective (2). We will introduce them in Section 4.1. The output layout with the lowest $q$ value (lower $q$ indicates better quality) is returned as the final output.

Datasets. We conduct experiments on 4 datasets, including RICO [27], PubLayNet [40], PosterLayout [9] and WebUI [24]. Their statistics and usages are illustrated in Table 3. For RICO and PubLayNet, we adopt the same dataset splits as LayoutFormer++ [15]. While for PosterLayout, the training set includes 9,974 poster-layout pairs, and the remaining 905 posters are used for testing. Regarding the WebUI dataset, we adopt the dataset splits provided by parse-then-place [24]. In all cases, the in-context exemplars are retrieved from the full training set.

Baselines. Since constraint-explicit layout generation tasks have task-specific and task-generic methods, we compare LayoutPrompter against both kinds of state-of-the-art methods on these tasks. Concretely, we choose LayoutFormer++ [15] as the common task-generic baseline. The task-specific baselines are (1) BLT [19] for generation conditioned on types (Gen-T), (2) BLT [19] for generation conditioned on types with sizes (Gen-TS), (3) CLG-LO [18] for generation conditioned on relationships (Gen-R), (4) LayoutTransformer [7] for completion, and (5) RUITE [30] for refinement. Moreover, we compare LayoutPrompter with DS-GAN [9] and CGL-GAN [42] on content-aware layout generation. We compare with Mockup [11] and parse-then-place [24] on text-to-layout.

Evaluation Metrics. To evaluate the performance of LayoutPrompter and baselines, we use the following quantitative metrics. For constraint-explicit layout generation and text-to-layout, we employ four standard metrics. Alignment (Align.) [22] gauges how well the elements in a layout are aligned with each other. Overlap [22] computes the overlapping area between two arbitrary elements in a layout. Maximum IoU (mIoU) [18] calculates the highest Intersection over Union (IoU) between a generated layout and real layouts. Fréchet Inception Distance (FID) [18] measures how similar the distribution of the generated layouts is to that of real layouts. Additionally, we introduce another metric Constraint Violation Rate (Vio. %) [15] to evaluate how well the generated layouts satisfy their input constraints. It is the ratio of violated constraints to all constraints. In text-to-layout, as a textual description may involve the type, position and size constraints of elements, we follow parse-then-place [24] and further break down this metric into Type Vio. % and Pos & Size Vio. %. As for content-aware layout generation, we adopt the eight metrics defined in DS-GAN [9]. Some of them belong to graphic metrics, such as Val, Ove, Ali, $\text{Und}_{\text{l}}$ and $\text{Und}_{\text{s}}$. Others are content-aware metrics, including Uti, Occ and Rea. Please refer to  [9] for more details.

Implementation Details. In this work, we conduct experiments on GPT-3 [3] text-davinci-003 model. We place $N=10$ exemplars in the prompt $\mathbf{P}$. For each test sample, we generate $L=10$ different outputs $y_{\text{test}}$. The hyper-parameters involved in the layout ranker module are set to $\lambda_{1}=0.2$,$\lambda_{2}=0.2$, and $\lambda_{3}=0.6$. When running GPT-3, we fix the parameters to the default values of the OpenAI API, where the sampling temperature is 0.7 and the penalty-related parameters are set to 0.

Tables 2, 4, 5 and Figures 4, 5, 6 show the quantitative and qualitative results on various layout generation tasks (see more qualitative results in Section C of the supplementary material). Although LayoutPrompter has not undergone model training and fine-tuning, the experimental results demonstrate that it can achieve comparable or even better performance than baselines, which proves that LayoutPrompter is a versatile and training-free layout generation approach. Below, we conduct a detailed analysis of the experimental results.

Constraint-Explicit Layout Generation. Table 2 shows the quantitative results. On each constraint-explicit layout generation task, LayoutPrompter is compared with a task-specific method and another common baseline, LayoutFormer++ [15]. Although not trained on these downstream tasks, LayoutPrompter still exhibits competitive quantitative results. Furthermore, it even outperforms the baselines on some metrics (e.g., Align. and Overlap on Gen-T task, RICO dataset). The corresponding qualitative results are shown in Figure 4. Here, we only compare with the state-of-the-art baseline (measured by quantitative metrics), LayoutFormer++. The qualitative comparison indicates that LayoutPrompter achieves as good controllability and generation quality as LayoutFormer++. First, the layouts generated by our approach satisfy various input constraints well, including type constraints, size constraints, relationship constraints, etc. Second, our approach can also produce visually pleasing layouts with well-aligned elements and small overlapping areas. Both qualitative and quantitative results demonstrate the effectiveness of LayoutPrompter.

Content-Aware Layout Generation. The quantitative and qualitative results are presented in Table 4 and Figure 5, respectively. Remarkably, LayoutPrompter surpasses the training-based baselines on almost all metrics. This indicates that LayoutPrompter is capable of producing higher-quality and more content-aware layouts compared to the baselines. The rendered results further validate the conclusion. For example, in columns (f) and (g) of Figure 5, the layouts from DS-GAN [9] contain serious misalignments and overlaps. And column (e) shows that DS-GAN sometimes fails to generate content-aware layouts. In contrast, our approach can not only produce aesthetic layouts but also avoid the most salient objects in the input canvas, such as the person, teddy bear, car, etc.

Text-to-Layout. The quantitative and qualitative comparisons are shown in Table 5 and Figure 6. Since text-to-layout is one of the most challenging layout generation tasks, LayoutPrompter slightly lags behind the current state-of-the-art method parse-then-place [24], especially on mIoU and FID metrics. However, on the other four metrics, LayoutPrompter is comparable to the baselines. Thanks to the excellent understanding capability of LLMs, our approach can better satisfy the constraints specified in textual descriptions in some cases. For example, in cases (d) and (e) of Figure 6, LayoutPrompter successfully generates 4-6 links and four logos, while parse-then-place makes wrong predictions about the number of elements.

Effect of Introduced Components. LayoutPrompter has three key components, including input-output serialization, dynamic exemplar selection and layout ranking. To investigate their effects, we perform the following ablation studies (see Table 6). (1) Since LayoutFormer++ [15] has proven the effectiveness of constraint sequences relative to other formats, we only study the effect of HTML representation, which is not covered in previous works. Specifically, we replace HTML with a plain sequence proposed by LayoutFormer++ [15] (denoted as w/o HTML) to represent the output layout. This results in a significant drop in FID and overlap metrics on Gen-T. (2) To understand the contribution of dynamic exemplar selection, we compare against its variant (w/o dynamic selection) that adopts random sampling for exemplar retrieval. LayoutPrompter achieves significantly better FID and mIoU across the board. Though the variant has better Align. and Overlap scores in some tasks, its noticeably poor FID and mIoU scores indicate that it fails to acquire the layout patterns in specific domains (e.g., the generated layout does not look like a real UI layout). (3) To understand the effect of the proposed layout ranker, we compare it against a variant (w/o layout ranker) that randomly picks a layout from model outputs. We find that the layout ranker consistently yields improvements on the mIoU and Align. metrics of all tasks.

Effect of Training Set Size. We switch training set sizes: 500, 2000, 10000 and full set (see Table 7). In our approach, the training set represents the exemplar retrieval pool. The results show that the performance of LayoutFormer++ drops rapidly as the training data decreases, but our method is much slightly affected. When training samples are limited (e.g., 500 and 2000), our approach significantly outperforms the training-based baseline on all metrics. These observations suggest that LayoutPrompter is a more data-efficient approach, which is effective in low-resource scenarios. Due to space limitations, more experimental results on stability, the effect of the number of examples, and generalization ability can be found in Section B of the supplementary material.