OmniFill: Domain-Agnostic Form Filling Suggestions Using Multi-Faceted Context

Research has investigated several ways for automating tedious or repetitive interactions in Web pages beyond suggesting or filling forms.

One line of work uses end-user programming to record demonstrations of Web page interactions, generalize these demonstrations into a program in a suitable DSL, and then let others replay this program later in a related but different context. Key tasks enabled by programming-by-demonstration have been sharing multi-step processes (e.g., Koala (Little et al., 2007), later CoScripter (Leshed et al., 2008)) and Web scraping (e.g., Ringer (Barman et al., 2016), Rousillon (Chasins et al., 2018)). A source of complexity in these systems is that Web pages may be interactive, their structure may be ill-formed, and they may change over time; more recent work has succeeded in using natural language processing techniques to make task specifications more robust and flexible (e.g. DiLogics (Pu et al., 2023)).

Some other approaches to automation rely on pixel-based reverse engineering of rendered interfaces (Prefab (Dixon and Fogarty, 2010), Sikuli (Yeh et al., 2009)), applying computer vision techniques to derive interface structure. The combination of DOM and visual features can surpass the limitations of these individual approaches (Kumar et al., 2013).

OmniFill is not addressing general Web automation, focusing instead only the narrower problem of suggesting text in Web form fields. However, it can draw on broader context than just prior demonstrations. While demonstrations are part of the context considered by OmniFill, the system also takes into account other sources of context explicitly and implicitly collected (e.g., text identified as important by the user), and world knowledge (as captured in the pre-trained LLM).

OmniFill is inspired by prior mixed initiave systems such as LookOut (Horvitz, 1999) and Citrine (Stylos et al., 2004), which also seek to reduce the tedium of manually completing forms with information that already exists in some other format. LookOut extracted information from email to pre-populate calendar events. Citrine parsed users’ copied text into typed fields using hand-constructed parsers for frequent content such as addresses, and could then auto-complete different address fields based on recognizing the structure of the form from prior demonstrations. Related interaction techniques such as Entity Quick Click also rely on entity recognition in copied text to accelerate copy and paste tasks (Bier et al., 2006). OmniFill extends this work by using an LLM and multiple sources of context to broaden the applicability of Citrine’s approach without requiring manually authored parsers for each type of data.

A number of technical approaches have been proposed to extract information from a larger document or corpus for the purpose of form filling, e.g. hidden Markov models (Borkar et al., 2000), or discriminative context free grammars (Viola and Narasimhan, 2005). Prior work using ML approaches has constructed direct mappings between observed user context and form fields using NLP techniques (e.g. (Hartmann and Muhlhauser, 2009)) or constructed domain-specific models of form dependencies (e.g. (Belgacem et al., 2023)). Because foundation language models, as used in OmniFill, can operate across many tasks and domains (Bommasani et al., 2022), they hold the promise of potentially obviating domain-specific recognizers and expanding task specifications beyond field-by-field extractions of suggestions from context.

Form filling is a special case of the larger problem of predictive text completion which seeks to predict, given some context of existing text, what text a user is likely to enter next.

Predictive Text Completion has existed for a long time in code editors (Teitelbaum and Reps, 1981), search query interfaces (Bast and Weber, 2006; Nandi and Jagadish, 2007), and predictive keyboards (Koester and Levine, 1994). More recently, language models have found widespread use completion suggestions in email composition (Goodman et al., 2022), and for larger code chunks (Nguyen and Nadi, 2022). In addition to research on the underlying technologies, researchers are also studying the impact of the use of predictive text interfaces on productivity (Ziegler et al., 2022; Palin et al., 2019), on the content of text being produced (Arnold et al., 2020), and on users’ perceptions of their tasks (Jakesch et al., 2023). While we restrict our focus in this paper to form filling, our conceptual framework and our multi-faceted context structure can potentially be applicable for analyzing a broader set of predictive tasks.

Researchers are increasingly investigating the utility of pre-trained language models for enabling new interactions. One key distinction is between systems that use LLMs to enable natural language based input on one hand; and systems that use LLMs as an enabling implementation technology for novel direct manipulation or other types of interactions.

As examples of the first category, Wang et al. show that LLMs are promising for enabling conversational interactions with mobile user interfaces (Wang et al., 2023), such as screen summarization and screen question asking. Stylette (Kim et al., 2022) enables re-design of Web pages through natural language commands. In the second category, user interactions are translated into appropriate prompts to a language model “behind the scenes” to offer better performance or novel capabilities beyond previous algorithmic approaches. Examples are TaleBrush (Chung et al., 2022), which allows users to sketch story arcs to condition the co-creation of stories with LLMs, and SayCan (Ahn et al., 2022), which uses LLMs to map real-world robot affordances to relevant situations.

OmniFill does not directly expose text interactions with the underlying LLM, instead offering a user-facing interaction closer to traditional brower autofill. Still, the use of an LLM as implementation technology enables the system to perform many natural language tasks that are implicitly used in form filling, such as extracting entities from context and making use of form field labels.

What information is being placed into the form? Where does it come from? In a predictive system, what information may be transcribed, transformed, or used as a retrieval key for the system’s current suggestion or completion?

We identify six broad classes of such information.

Historical user behavior:

What information has the user entered into the form or similar forms in the past? Example: browsers offer dropdown suggestions for recognized fields, even in previously-unseen forms.

Explicitly-foregrounded information:

Users may explicitly call attention to information outside the target form interface. Example: importing a CSV into a system, or the “select, then copy” operation of a clipboard.

Implicit browsing context:

User activity may provide additional context relevant to the task even if not explicitly foregrounded. Example: scrolling through the subject lines of an email inbox.

Current form state:

Some form filling operations make use of information already present in the form. Example: creating a username based on the values of a “First name” and “Last name” field.

External general knowledge or language knowledge:

Some information inserted into forms comes from the surrounding world, especially in combination with other information sources. Example: after typing “Paris” in a “City” field, a system may offer “France” for the “Country” field.

User-specific external knowledge:

User-specific information may come from sources other than form-filling activity or recent browsing context. Example: a system may offer completions based on the data from the user’s phone contacts.

What transformations must be applied to inputted information by either the user or a predictive system making field suggestions?

Exact transcription:

An entire input is transcribed verbatim into the output field. Example: a user copies a URL into their clipboard, and their mobile browser address bar offers to “paste” that full URL.

Literal extraction:

Information (in any of the forms described in Section 3.1) is transcribed verbatim into the target form but must first be extracted from some larger source.

Format transformation:

Information must be transformed into a different format. Example: a task requires a name to be converted from all-caps (as it appears in the source) to title case (as required by the target form).

Quantitative transformation:

The task demands arithmetic operations to be performed on available information before inserting into the target form.

Semantic transformation:

The task requires a transformation of the available information that produces an output that is syntactically and structurally distinct from the input information but still semantically related. Example: upon viewing an email reading, “I don’t eat meat, but I can eat animal products like cheese”, the user might write “Vegetarian” into the target form to match the format of other form responses.

When the task consists of multiple instances of a form being filled, how does the task vary between form fills? Once the task is well-specified, how flexible does that specification need to be to handle the source and target structures?

Fixed source structure:

Does information used to derive field values come from the same place, in the same format, from case to case? An example of a task with fixed information source structure is one in which information is sourced from a spreadsheet and each row of the sheet is used to construct a submission in the target form.

Varied source structure:

An example task with varied information source structure is the information-gathering task from our user study, in which users repeatedly fill the same form but with information from a differently-structured website for each submission of the target form.

Fixed target structure:

Is information always inserted into the same field of the same form, or can each submission to a form take a different structure? Any task involving repeated submissions to the same form has a fixed target field structure.

Varied target structure:

An example of a task with varied target field structure is filling out many distinct job applications; although the information being filled is mostly the same from submission to submission, the form structure (and in some cases, the demanded information format) changes each time a form is filled.

What information is available that specifies the process of completing the task? If a user is completing the task manually, how much understanding of the task might be gleaned by someone looking over their shoulder? These specifications are distinct from the information demands described in Section 3.1, which refers to information that may be actually entered into the form (potentially after a transformation) or used to reference other information to be entered into the form.

Implicit specification:

Task specifications may be implied by the structure of collected context or target form. For example, a user who copies an address and visits a form with an address field likely intends to insert that address into the target form. This structural information may take a machine-readable form (as with the autocomplete attribute in HTML) or natural language forms (as with text description labels on fields).

Instructions:

Tasks can be specified with form-filling instructions, present in, for example, the form or browsing context. Instructions may be precise and machine-readable, as with tasks specified using code or macro software. They may also come in the form of natural language, which may leave gaps in the specification.

Examples:

Many tasks can be approximately specified using a number of examples, as demonstrated in the programming-by-example literature. When prior examples of user behavior are available, approximate task specifications can be inferred and realized in synthesized code. Examples may also help to clarify ambiguities in vague instruction-based specifications.

OmniFill consists of two local components, a browser extension and local server both implemented in TypeScript, and a connection to a remote LLM server with a “chat completion” generation API.

Together, these components enable a streamlined interaction flow, depicted in the system screenshots in Figure 4, offering users low-friction interactions to foreground relevant context, invoke OmniFill’s backend to offer completion suggestions, and show the system examples of the form-filling task. This all occurs in situ, requiring no task-specific configuration and minimal interaction outside the context sources and target form already in use.

To allow the system to make useful inferences in varied situations, OmniFill constructs a prompt containing three main facets of task context (visualized in Figure 6):

• •

  Current browsing context, representing context explicitly inserted into OmniFill’s scrapbook by the user as well as implicitly-observed Google searches.

• •

  Form structure state, including the inferred name for each form field, the initial value of each form field, and any edits made by the user to each form field’s value.

• •

  Prior examples saved by the user, each including browsing context, form structure, and the intended final form state before saving the form.

This prompt structure is designed to support many broad tasks without requiring task-specific configuration. Some tasks may primarily use only one or two of these context facets, and others may combine information from all three. Depending on the task, these context facets may all contain information demands of the task, and they may all contain information that helps OmniFill infer and carry out a task specification, so all three facets are included in each prompt to the LLM.

Participants joined a Zoom call and completed two distinct tasks with OmniFill through a remote virtual desktop, with a demonstration of the system incorporated into the first task. In the first task, participants were instructed to “forage” information (Pirolli and Card, 1999) from websites of schools, inputting that information into a structured form (an instance of EspoCRM (esp, 2023)), shown in Figure 7. First, participants were directed to the website of a public high school and asked to spend a few minutes to fill the form manually, without using OmniFill. For this test website, participants who took longer than a few minutes were offered assistance locating the form data.

Then, participants were shown a demonstration on a different school’s website, using OmniFill to fill the same form. In this demo, participants were shown how to use the Alt+select interaction to add context to their scrapbook, then shown how to generate and accept suggestions for the form. Participants were shown that selections can be made approximately, and that it was possible (but not necessary) to make multiple distinct selections before returning to the target form. Participants were shown how to add context to the scrapbook manually, for situations where the Alt+select interaction failed (e.g. when viewing PDFs). In the demonstration, participants were shown and reminded that form inputs may be typed manually and need not come from OmniFill suggestions. We also checked OmniFill’s “Automatically save examples for this site” checkbox so that the form’s “Save” button would trigger the current form inputs to be saved as an example.

After the demonstration, participants were asked to practice using the system by again filling the form with data from the website they had already manually collected information from. Then, participants were given approximately 15 minutes to perform the task freeform, finding websites for other schools and filling out the form, one school at a time. To encourage participants to find websites with varying structures, we asked them to research schools from different school districts. We also asked participants not to spend more than a few minutes on any one website and indicated that it was okay to leave some fields blank if they didn’t think they would find the information.

The data-formatting task required OmniFill to recognize patterns when editing forms with prepopulated data.

Participants were shown a mock “human resources” website (an instance of Admidio (adm, 2023)), pre-filled with fake company membership data. We prepopulated every user profile with a T-shirt size, most profiles (45 of 51) contained a mobile number, and some (14 of 51) contained a home phone number. The string format of the shirt size and phone number were randomly chosen for each member profile, as depicted in Figure 8. Each participant received the same mock company data, but the order of the data was randomized so that OmniFill would behave differently for each participant. No participant saw more than 28 of the 51 total member profiles during the ten minutes allotted to perform the task.

Participants were shown a series of instructions for their task, asking them to update each phone number in the member profile (if present) to a fixed format and to update each T-shirt size to one of a few fixed options (depicted in Figure 8). These instructions were shown in a screenshot so that participants could not Alt+select them, since we wanted to investigate OmniFill’s pattern recognition; although participants could have manually typed the instructions into the scrapbook, none did. Then, we instructed participants to visit each member profile in order, updating the phone number and T-shirt fields following these instructions, with OmniFill automatically saving examples each time the form was saved. We also instructed participants to click the “Suggest with OmniFill” button in the extension sidebar and wait for its response on each profile before making any edits, testing the system’s ability to recognize the task specification and offer the correct edits as suggestions. We advised participants that they were not required to accept OmniFill’s suggestions or use its suggestions unchanged.

After completing both tasks, participants were asked briefly about their experience in a conversational interview, including questions about OmniFill’s perceived utility, concerns about real-world use, confidence in the system’s accuracy, and confidence that they would notice OmniFill’s mistakes.

Participants readily used OmniFill in the information-gathering task, accepting OmniFill’s suggestions for almost all form fields in each school contact they saved; seventeen participants, when asked, said they would likely use OmniFill if they had to perform the information-gathering task in real life. Although suggestions for this task largely used information from the current browsing context portion of the prompt, even this task was able to benefit from OmniFill’s multi-faceted prompt. For example, participants who typed schools’ country name as “United States” tended to receive this as a suggestion, where participants who preferred “USA” often received the suggestion in this format, indicating that OmniFill was making use of prior examples in its prompt.

In the data-formatting task, OmniFill proved more successful for some participants than for others, and when the system did eventually succeed more consistently, many examples were often required for this to settle. Figure 9 visualizes OmniFill’s success suggesting values for the phone number fields in particular. Each row represents a participant, and each box represents a completion request sent through the system. Only the first request is shown per member profile, even if participants invoked the system more than once, and we do not show requests for profiles where no phone number change needed to be made. In this figure, the box is shown in green only when both phone number fields are suggested correctly (although in many cases there was only one phone number, so only one change needed to be made). Cases where users accepted an incorrect suggestion or rejected a correct suggestion (choosing instead to type the phone number manually) are also shown.

After conducting the study, we also called the GPT-4 API (OpenAI, 2023) using the same prompts as those used in the study to observe system performance with the more powerful model, shown on the right side of Figure 9. Because GPT-4 has a smaller context window (8,192 tokens) than gpt-3.5-turbo-16k, some prompts could not be run for this analysis; those are depicted with gray stripes. In addition to the context window size, real-world trade-offs of using GPT-4 for OmniFill include cost and latency; for these reasons and to better understand users’ behavior with a less-effective model, we used GPT-3.5 for our studies.

Compared to the meticulous copying-and-pasting (or memorizing-then-typing) strategy participants employed to fill out the information-gathering task’s contact form before being introduced to OmniFill, participants’ information-foraging behavior during the free-form component of this task was approximate and opportunistic. We observed participants collecting information for OmniFill’s bag of context quickly and even haphazardly, as one participant put it, often with little regard for the later extraction step.

Because OmniFill assumes the role of locating relevant information in its bag of context during the information-gathering task, users who want to collect context opportunistically and avoid keeping track of the bag of context must place trust in OmniFill’s inferences. We observed that users often trusted OmniFill’s suggestions, even when that trust may have been unfounded.

The contact form contained a “grade levels served” field, which should be filled with e.g. “9-12” for a typical American high school. When websites did not explicitly include this information, some participants were reluctant to make assumptions for the value of this field. Although OmniFill often withheld suggestions unless the relevant information was explicitly present in its bag of context (for example, we never observed the street address field being incorrectly populated), the model did often return with a suggestion for the grade level field without the context including this information; this happened at least once for 15 of 18 participants.

It was rare, however, for participants to question this information when OmniFill suggested it; only one participant did explicitly reject this overeager grade level completion, and that participant had selected only a very small amount of text that they could immediately observe did not contain the grade level information. In a flipped example, one participant, who had filled out most of the contact form, was struggling to find the name of the school’s principal. They chose to select the name of an assistant principal for their scrapbook. Only after heading back to the contact form and observing that OmniFill did not offer a suggestion for the “Principal” field did the participant give up on finding the information and move onto the next school.

Although the data-formatting task contained fewer opportunities for participants to rely on OmniFill to make “judgment calls”, we still observed two instances of participants accepting and saving OmniFill’s incorrect suggestion. For each of these two participants, OmniFill made the strange suggestion in one case to duplicate a correctly-formatted phone number into a blank phone number field. Although both participants had already correctly handled this case in the past, they chose to accept this suggestion from OmniFill; one said aloud, “Oh, yeah, sure. Why not?” as they did this, indicating a willingness to allow OmniFill to take the reins in deciding and carrying out the task specification.

OmniFill was not perfect, occasionally visibly failing to extract explicitly-foregrounded information in the information-gathering task or offering incorrect suggestions (or no suggestions at all) for the data-formatting task.

Still, this partial success offered a value-add in both tasks. In the information-gathering task, participants could always fall back to traditional information-foraging techniques to find data that OmniFill failed to extract. In the data-formatting task, OmniFill’s success rate on offering correct T-Shirt sizes was high (across the 18 participants, OmniFill offered a correct suggestion in 306 of the 347 cases (= 88%) where the T-Shirt size needed to be updated), so participants were often presented with at least one correct suggestion even when the system couldn’t get every field right.

Even when the phone number suggestion was incorrect, we found that participants often accepted the incorrect suggestion and then made changes to the field, rather than editing the original text field value. Of the 16 participants who received a suggestion to change a phone number from one incorrect format into another incorrect format, 15 accepted one of these suggestions at least once (and 9 did this multiple times); Figure 9 indicates incorrect suggestions which were accepted by the user. For example, a common failure mode for this task was for OmniFill to format the phone number correctly but without removing the country code (i.e. +1 (###) ###-####) or to hyphenate a ten-digit number (transforming ########## to ###-###-####). By accepting these suggestions, participants allowed OmniFill to get them closer to their desired result for the form field.

Many participants noted that partial failure may not be harmless, however. Ten participants, when asked about concerns using OmniFill in the real world, cited accuracy considerations. Many noted, either during the interview or while completing the tasks, that although they may notice “obvious” errors or those that don’t conform to the task specification, they might not notice if OmniFill inserted incorrect information in the information-gathering task or changed phone numbers in the data-formatting task. Four participants, including some concerned about OmniFill’s accuracy, still suggested that the potential for human error in a task like phone number formatting was high, and that they had more trust in OmniFill to be correct.

When explicitly asked about real-world concerns about using OmniFill, only six of 18 participants mentioned privacy or information security as a consideration, even though the data-formatting task involved working with simulated personal information. We did not explain to participants upfront that OmniFill does not run entirely on the user’s local computer, but even users who know they are interacting with an online system may not consistently protect the privacy of their information (Sundar et al., 2013). Systems that allow users to ingest other peoples’ personal information must be considered even more strictly. A system like OmniFill benefits from passively observing the user’s behavior or collecting implicit context (e.g. bringing in text immediately before and after a user’s text selection in case their mouse cursor “aim” was imperfect), but this requires trust in the remote LLM API used by the system. Because OmniFill is designed to interoperate between siloed ecosystems, privacy concerns persist even outside of systems that are known to track user behavior.

Running general suggestion systems locally on the user’s computer may become feasible as model architectures grow more efficient and computers become more powerful, keeping model queries private. Still, future work may choose to store many prior user examples (more than what can fit into a single LLM prompt) for later retrieval during prompt construction (as in (Levy et al., 2023)). System designers should be careful when collecting and opaquely storing browsing context long-term, even locally, since this practice can increase the consequences of a system breach.

Even after the system appears to have “learned” a task, the lack of a rigorous task specification can cause occasional errors (as Figure 9 demonstrates in the GPT-4 section). OmniFill is designed to assist users in their manual tasks, not construct reliable automations. However, “automation bias” is known to result in uncertainty among users of automated systems (Bond et al., 2018), and a “good-enough” automation of menial tasks may cause the user to “check out” and stop manually verifying the results of suggestions, as reported by some participants after completing our data-formatting task. As Figure 9 illustrates, success can be highly dependent on choice of model, depending on the task at hand. This suggests challenges for developers building systems that are not entirely under their control.

Since users cannot inspect the LLM’s workings, it is difficult to form a mental model of which types of errors the LLM is likely to make, meaning even partial success of the system may have variable utility. We observed users accepting OmniFill’s “judgment calls” even though the system had never been told how it should make these judgments; the “authority” of the system seemed persuasive even in those very situations where an automated system cannot know how to behave.

Future improvements to such systems may further reduce the cognitive load demanded of users during certain tasks (e.g. by improving implicit context collection so that less information needs to be explicitly foregrounded), but these exact improvements may cause users to be less likely to notice when the model makes an error. Our results from the information-gathering task demonstrate that the system is already powerful enough to permit users to collect and use information without noticing that they had collected it. How, then, would a user in this situation notice if the suggestion is incorrect?

Context visibility may play a key role. Because the “collect-then-fill” strategy recruits OmniFill’s bag of context as an auxiliary “memory” for the user, future work in constructing interface features could allow users to peer into the bag of context and view context sources in situ. Since information used in suggestions is often present in the bag of context, either in some literal or pre-transformation form, these features could surface just the context that matches with OmniFill’s suggestions. Although a literal search for each suggestion in the bag of context may not yield results when the task involves transformation operations, additional prompts to a lighter-weight LLM (or a semantic search using an embedding model) could be used to handle many simpler cases. OmniFill’s full LLM prompt would still be responsible for producing the actual suggestions, since the full bag of context may be valuable for making high-quality predictions (e.g. if the form structure is poorly-labeled but can be learned through prior form-filling examples), but post hoc “attribution” may be achieved through an auxiliary system.

Although in situ training of the system can be a low-friction way to offer predictive suggestions, systems should provide users with the ability to view and refine their task specifications, e.g. by curating their set of examples to maximize system accuracy. Future work could assist users in this process by detecting and surfacing potentially anomalous prior examples or by engaging the user in a dialogue to define and fine-tune task specifications as the system is used over time.