Visual Embedding of Screen Sequences for User-Flow Search in Example-driven Communication

Communication of UX considerations to stakeholders is critical in collaboration for user-centric product design (Gray, 2016; Rose and Tenenberg, 2016), helping product managers and developers embrace user-centric decisions (Shukla et al., 2024; MacDonald et al., 2022). Effective communication addresses two major collaboration challenges: keeping pace with collaborators and aligning priority conflicts (MacDonald et al., 2022). It ensures the UX is sufficiently considered among collaborators in the fast-paced product development process (Holtzblatt and Holtzblatt, 2014). Also, it aligns UX considerations with other collaborators’ priorities (e.g., retention and conversion metric) (Shukla et al., 2024). To support communication, previous research explored the strategies including rhetoric techniques (Rose and Tenenberg, 2016; Dumas et al., 2004; Gray, 2016), exposing stakeholders to a user (Følstad et al., 2012), and visual illustrations (Hornbæk and Frøkjær, 2005; Rose and Tenenberg, 2016; Gray, 2016; MacDonald et al., 2022) to demonstrate the significance of the UX considerations.

Example-driven communication is one of the effective communication strategies (Karlgren and Ramberg, 2012). In design practices, examples inspire practitioners by showing how content structures and integrates (Lee et al., 2010; Swearngin et al., 2018; Choi et al., 2024). Similarly, when collaborating with stakeholders, examples strengthen the communication by allowing empirical observation (Herring et al., 2009; Kang et al., 2018). They clarify the rationale behind considerations (Kang et al., 2018) and help establish a shared understanding of problems (Karlgren and Ramberg, 2012). While examples take various forms, a prominent theme is user flow, incorporating journey maps, video/audio reels, flows, and user stories (MacDonald et al., 2022). User flow represents semantics that arise from interaction sequences such as user task (Deka et al., 2016), which is more challenging to infer from than static screens. In response, this work addresses the computational method to help practitioners search user flow examples through semantic embedding.

Semantic embedding of screens, which produces computational representations received continuous focus for downstream tasks such as search (Huang et al., 2019), screen captioning (Wang et al., 2021), component prediction (Li et al., 2021b; Jiang et al., 2024), and code generation (Moran et al., 2018). A mainstream approach leverages view hierarchy and layout information as representative features (Deka et al., 2017), demonstrating embedding models’ ability to identify visually coherent examples (Deka et al., 2017; Huang et al., 2019). Studies further add semantic tags for each layout components (Li et al., 2021b; Bunian et al., 2021; Liu et al., 2018; Wu et al., 2021b) or text and images with pre-trained encoders (Wang et al., 2021; Ang and Lim, 2022; Li et al., 2021a; Baechler et al., 2024; Bai et al., 2021), effectively capturing domain-specific screen characteristics like app category or feature (Wang et al., 2021; Ang and Lim, 2022; Li et al., 2021a; Bai et al., 2021). Especially, Wu et al. incorporated temporal dimension in embedding to capture the screen animation (Wu et al., 2020)

Furthermore, studies explored methods to embed user flows through screen sequences, which represent a meaningful task through step-by-step interactions (Deka et al., 2016). Embedding screen sequences expands computational capabilities including user flow search (Deka et al., 2016), screen relation inference (He et al., 2021; Li et al., 2021b), and interaction prediction (Rawles et al., 2024). The primary approach involves learning temporal context with order-sensitive models (He et al., 2021; Li et al., 2021b), and studies explored large language models’ capability to comprehend user flows (Lu et al., 2024; Chen et al., 2024). Recently, Yicheng et al. proposed a JEPA-based design (Assran et al., 2023) that generates user flow video embeddings to represent user intent (Fu et al., 2024). Building on recent video understanding models (Zhao et al., 2024; Arnab et al., 2021; Yuan et al., 2023), we address this problem with order-sensitive multi-head attention pooling of visual features in screen sequences.

We recruited four UX practitioners with professional experience through snowball sampling. The group consisted of three product designers (N = 3) and one UX researcher (N = 1), with an average of 1.27 years (SD = 7.8 months) of professional experience. To encourage detailed responses, the group interview was conducted in the interviewees’ primary language. Two researchers collaboratively analyzed the study data through thematic analysis (Saldana, 2021).

The study began with a five-minute introductory session of the study protocol and consent. The main interview was divided into two sub-sessions, each lasting around 40 minutes. In the first sub-session, we focused on (Q1) identifying the context of example-driven communication. We asked questions about interviewees’ current collaboration patterns, existing communication challenges, and strategies they used to address these challenges. In the second session, we focused on (Q2) understanding the important attributes of examples. We introduced a hypothetical AI support tool scenario that provided examples to interviewees and asked them for evaluation. The researchers ended the interview by summarizing the discussion and asking for additional feedback.

To enable semantic embedding of screen sequences, we design an encoder-pooler head structure outlined by Yuan et al. (Yuan et al., 2023) in which the pooler aggregates information from encoders. The pooler head is a single cross-attention layer, capable of attending to meaningful visual features through a single learnable query token and self-attention. Our model captures the key visual features relevant to user flow-related semantics of the screen sequence. Additionally, we add padding and masking in the pooler to allow the embedding of variable-length screen sequences.

Visual Encoder. (Figure 2A) To capture visual features from each screen, we employ DinoV2 (ViT-L/14) with registers (Darcet et al., 2023; Oquab et al., 2023) as a frozen image encoder that takes 224$\times$224 screen images and outputs 1024-dimensional feature embeddings. Although end-to-end fine-tuning improves the performance (Yuan et al., 2023), we assume that ViT can capture the screen semantics without fine-tuning based on an observation with CLIP (Radford et al., 2021) by Seokhyeon et al. (Park et al., 2023).

Pooler Head. (Figure 2B) After the visual encoder extracts per-screen embeddings, we feed each sequence of embeddings into a learnable pooler head that produces a single vector representation. When feeding in, to handle variable-length sequences, we pad shorter embedding arrays to a uniform length and use masking to make the pooler focus only on valid frames. We base our pooler head on a single-layer multi-head attention pooling mechanism (Yuan et al., 2023). The model first performs linear projection of each 1024-dimensional screen embedding into a 256-dimensional space and adds a learnable positional embedding to reflect the temporal order. Next, a single query token attends to all positions in the sequence via multi-head attention (MHA). The input and aggregated output are normalized, passed through a small multi-layer perception (MLP), and finally projected to 1536 dimensions to match the size of text embeddings for user flow description. This single-query design allows the pooler head to understand the underlying spatial-temporal relationship between screens in a screen flow.

We train our model using data from the Android in the Wild (ATIW) (Rawles et al., 2024), which provides extensive sequences of real-world device interactions. In particular, we focus on a subset of the “SINGLE” partition, which focuses on the single-user flows. While keeping the variable length of each sequence, we filter sequences to those with 3–6 screens each, resulting in 12,659 final sequences (episodes) and a total of 49,590 screens. Our motivation is to isolate a moderate range of screen lengths often encountered in single-user flows (e.g., a few steps to add an item to a cart) and reduce the variability within the dataset. We did not include the visual interaction traces between screens in the image. Each screen image is resized to 224$\times$224 pixels during preprocessing through non-uniform scaling to match the input constraints of DinoV2 (Oquab et al., 2023). Additionally, we used OpenAI’s text-embedding-3-small (OpenAI, 2024), to generate 1536-dimensional embeddings for each sequence’s task description provided by ATIW.

Our pooler head is trained to align screen-sequence embeddings with text embeddings through a contrastive learning objective similar to CLIP (Figure 2C) (Radford et al., 2021). Specifically, we adopt a method suggested by Zhao et al. (Zhao et al., 2024) in which the model minimizes a symmetric cross-entropy loss over the similarity scores of all image sequence-text pairs in a mini-batch:

where $\mathbf{V}$ and $\mathbf{T}$ denote mini-batch matrices of normalized screen-sequence and text embeddings, and $\tau=0.07$ is a fixed temperature. For training, randomly divide our dataset into training (90%) and validation (10%) splits, and train the model for 100 epochs using the Adam optimizer at a learning rate of $1\times 10^{-4}$, with a batch size of 1024 episodes per iteration. During each gradient update, the pooler attends the visual embedding of screens and produces a single output, which is then compared against the corresponding textual embedding via the contrastive loss. Since the visual and text encoder remain frozen, the network converges quickly. We observed that the validation loss stabilized at approximately 100 epochs with a value of 2.616 (using PyTorch seed 123).

We recruited 21 participants—14 with UX experience and 7 without—for an online survey that included 27 multiple-choice questions and 15 optional open-ended questions. The study consisted of two main tasks.

In Task 1, participants assessed the similarity between a source screen sequence and two candidate sequences; each candidate was obtained using the baseline model and our model. We randomly sampled five source screen sequences. We measured the similarity using a 5-point Likert scale across four dimensions (service similarity, screen type similarity, content similarity, and visual similarity). These dimensions reflect established practices for assessing screen-based relevance and layout coherence (Li et al., 2021b). To minimize potential order bias, we randomized the presentation order of two candidate sequences. In Task 2, participants assessed design examples retrieved by our model in practical scenarios (e.g., ”designing a search result page”) from a UX practitioner’s perspective. Based on established design applicability measures (Jeon et al., 2021), participants rated each candidate example on five aspects using a 5-point Likert scale: layout alignment, UI component utility, innovation potential, integration feasibility, and overall reference value.

For data analysis, we examined the responses of both tasks using different analytical approaches. For Task 1, we verified the normality of the data distribution using the Shapiro-Wilk test ($p>0.05$), confirming normality, and subsequently conducted paired t-tests to statistically compare our approach with the baseline. For Task 2, we employed a descriptive analysis of the quantitative Likert-scale response.

The evaluation result suggests that visual information can effectively reinforce user flow semantics. Our model achieved statistically higher ratings consistently, indicating a stronger alignment with human perception of similarity. Notably, despite the statistical significance and consistent gap, the average rating difference between our model and the baseline was relatively moderate. This indicates that the visual features captured by our model consistently add the information available in the textual description that predicts the semantic similarity between the two sequences. Specifically, we observed that screen sequences retrieved by our model shared a series of visual features with the source screen sequence, and these features add precision to the task description (e.g., the keyword “search” retrieved screen sequences that contain interactions with a search bar). Considering the model’s simple architecture, the model’s capability to link textual semantics to a specific sequence of visual features is promising.

Based on our formative study, our sequence-based approach demonstrates the potential to enhance example-driven communication in UX by providing rich user journeys, extending beyond isolated examples (Karlgren and Ramberg, 2012). Regarding this, features that connect sequence examples with journey maps or task flows could enhance contextual understanding (Karlgren and Ramberg, 2012; MacDonald et al., 2022). Additionally, improving system explainability would boost trustworthiness as UX communication has an evidence-based nature (Herring et al., 2009; Kang et al., 2018). Furthermore, for the effective utilization of retrieved sequences, careful integration with existing design tools such as Figma or Sketch would be recommended (Figma, 2025; Sketch, 2025).

Another notable finding was the UX search expectation between convention and innovation. Our sequence-based search has the feasibility to help identify established UI patterns. Furthermore, the user preference for familiar UI conventions reflects practical reference needs in professional settings. However, to support diverse practical scenarios, supporting innovative design exploration for diversity aspects can be considered for additional search options by providing diversity metrics rather than similarity alone.

Despite these promising results, considerable uncertainties regarding the model’s performance and applicability remain. Our model is trained on a narrow subset of the ATIW dataset (Rawles et al., 2024) with domain-specific examples, hence the ability to handle diverse user flow types remains unknown. The model’s understanding of textual descriptions is limited to the label distribution in training data, thereby constraining the selection of search queries. Additionally, the current architecture uses a relatively simple attention-pooling method with a pre-trained ViT and text encoder, focusing primarily on evaluating the model’s capabilities. To advance this approach, future iterations could incorporate screen text (Wang et al., 2021) and user interactions between screens to model training. Incorporating model design strategies such as variable resolution input (Lee et al., 2023), low-rank adapters (Yuan et al., 2023), and masked autoencoders (Tong et al., 2022; Zhao et al., 2024) could further enhance the performance. Integrating language decoders (Li and Li, 2022; Baechler et al., 2024) could extend functionality beyond search to tasks such as interaction localization, question answering, and action prediction with relevant benchmark evaluations. Similarly, a comparative evaluation with an existing video foundation model (Zhao et al., 2024; Wang et al., 2022) and previous screen sequence embedding approaches (Deka et al., 2016; Li et al., 2021b; Lu et al., 2024) against benchmark can highlight the relative benefit of our approach. Furthermore, an evaluation with a more diverse UX practitioners can expand exploration towards more specific design scenarios in real-world applications.