ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming

Searching visual scenes for specific pieces of information has always been an important aspect of assistive technology design. In early remote human assistance approaches like VizWiz, users would submit a question along with an image, and assistants would use their human intelligence to determine a relevant answer (Bigham et al., 2010b). This need for specific information is present across a variety of accessibility contexts: Find My Things and Kacorri use teachable object recognizers to help blind users locate specific possessions (Morrison et al., 2023; Kacorri, 2017), VizLens helps blind users search for specific buttons on physical interfaces (Guo et al., 2016), and CueSee highlights products of interest for people with low vision (Zhao et al., 2016). Even outside of accessibility contexts, the Ctrl-F shortcut for ‘find’ is ubiquitous. General-purpose and specific assistive technologies each have their uses; compare the ambient audio cues in Microsoft’s Soundscape (Microsoft Research, 2021) to navigation directions from Google Maps, neither is a direct replacement for the other.

Yet, current automated assistive applications present challenges to getting specific information quickly. Whether they run on a live camera feed (e.g, Seeing AI (Microsoft, 2021)) or on a static image (e.g, the GPT-4 powered ‘Be My AI’ (Eyes, 2024)), commercial applications have taken a general approach to describing visual information, conveying all results from the underlying OCR or object detection models, or generating as rich of a description about the visual content as possible. While this is sometimes desirable, it risks slowing down and increasing the cognitive burden on users who are looking for something specific (Gamage et al., 2023). In this work, we aim to target this need for specificity. ProgramAlly is a live assistive technology that can provide continuous feedback, but it also aims to capture a user’s explicit intent through the creation of filtering programs.

In accessibility research, personalization of technology to meet user needs is used to reduce the burden of accessibility on users (Garrido et al., 2012; Sloan et al., 2010; Stangl et al., 2021). This typically leaves the function of the technology unchanged, but aims to automatically map the input and output mechanisms to new systems or modalities (Gajos et al., 2007; Wobbrock et al., 2011). For example, Yamagami et al. recently considered how people with motor impairments would create personalized gesture sets that map to common input mechanisms (Yamagami et al., 2023).

Work customizing the functionality of assistive technology is more limited in comparison. In AI assistive technology, teachable object recognizers have been used to allow users to personalize recognition models themselves (Kacorri et al., 2017). Users capture their own image or video data of unique objects that can be stored and later recognized (Theodorou et al., 2021; Morrison et al., 2023). These approaches can be more useful than off the shelf object recognition models as they are customized to user’s specific needs (Kacorri, 2017; Bragg et al., 2016). However, for commercial applications, users have limited avenues for customization. While screen readers can be personalized through a variety of settings, shortcuts, and add-ons (Momotaz et al., 2021), AI powered assistive applications are typically part of closed software ecosystems. While they may have some settings within the application for things like language and output speed, this is typically the extent of the customization. Through this work, we hope to demonstrate new methods for personalizing assistive technology functionality to meet unique user needs.

DIY communities have adopted an approach to making centered around personalization, democratization, and collaboration (Kuznetsov and Paulos, 2010; Tanenbaum et al., 2013). For assistive technology, DIY approaches can help to address assistive technology adoption due to unique or changing needs (Hurst and Tobias, 2011). To this end, prior research on DIY assistive technology has sought to make the process of prototyping and making more accessible to participants with a range of technical skills (Meissner et al., 2017; Rajapakse et al., 2014).

Most of this research focuses on making physical tools for accessibility (e.g., making 3D-printed devices like an ironing guide, right angle spoon, or tactile graphics (Buehler et al., 2015), prototyping custom prosthetics (Hofmann et al., 2016)), rather than software tools. Some tools are being developed to support blind people in DIY-ing more high-tech hardware sensing systems, such as A11yBits (He et al., 2023) or the Blind Arduino Project (Institute, 2024). While these raise the ceiling of high-tech DIY creation, little research has focused on DIY-ing new software systems for existing devices users already own. In their original case studies of DIY assistive technology, Hurst and Tobias highlighted one instance of ‘high-tech custom-built assistive technology’, wherein a team of professional programmers worked with an artist with ALS to create software that used eye-tracking input for drawing (Hurst and Tobias, 2011). Lowering the barrier to entry for creating technically complex assistive software is an important next step in enabling people to DIY personally meaningful assistive technology.

Decades of end-user programming research has sought to understand and support programming work done by people who are not trained as programmers (Myers et al., 2006). While initially focusing on end-user programming in professional contexts (e.g., using spreadsheets or other domain-specific tools (Scaffidi et al., 2005)), a variety of approaches have been developed to support programming for personal utility as well. For example, Marmite is an end-user programming tool that allows users to create new applications by combining data and services from multiple existing websites (Wong and Hong, 2007). Here, we describe previous end-user programming approaches that work towards the goal of making programming more approachable for novices. In this work, we aim to apply these existing end-user programming approaches to the domain of visual assistive technology, enabling blind people to have a new level of control over assistive software.

Block-Based Programming. Visual, block-based programming approaches allow users to create programs by graphically organizing elements. These approaches often aim to support novices by providing pre-structured statements to reduce or eliminate syntax errors (Mohamad et al., 2011), for example, as in Scratch (of Technology, 2024). While these approaches are commonly used in educational settings (Weintrop, 2019), they are also used in commercial mobile automation tools to provide sets of components that users can arrange as they wish to create time-saving automations, as in Shortcuts on iOS (Apple, 2022) and Google Assistant (LLC, 2022).

Natural Language Programming. Further work has aimed to synthesize programs from natural language alone. These approaches commonly require a set of training data consisting of queries and desired automations (Desai et al., 2016; Li et al., 2020). Large language models have also been used for program synthesis, with mixed results (Austin et al., 2021; Vaithilingam et al., 2022).

Programming By Example. Programming by example approaches alternatively allow users to create programs by providing demonstrations of desired functionality, without the need for any code (Lieberman, 2001; Cypher and Halbert, 1993). Programming by example has been implemented in a range of domains, for instance, Rousillon automates web scraping with a demonstration from users on how to collect the first row of a data table (Chasins et al., 2018), and Sugilite automates actions on mobile interface using a demonstration and natural language request (Li et al., 2017).

End-User Programming and Accessibility. The accessibility of programming tools is a nascent area (Potluri et al., 2022, 2018; Pandey et al., 2022) that has largely focused on developers rather than end users. End-user programming approaches have occasionally been applied to accessibility contexts for the purposes of sharing accessibility bugs and teaching blind children. For example, for web accessibility, demonstration has been used as a method for end-users to convey accessibility issues to developers (Bigham and Ladner, 2007; Bigham et al., 2010a). Story blocks is a tangible block-based tool for teaching blind students programming concepts (Koushik et al., 2019). We aim to continue and extend this line of research by supporting blind end-user programmers in creating new visual assistive software.

Overall, we designed ProgramAlly based on three primary goals: Expressiveness, Approachability, and Accessibility.

D1: Expressiveness. ProgramAlly’s goal is to be an interface where users can customize off-the-shelf models for their own uses. Programs should be able to support a wide variety of real-world use cases through a flexible structure and range of models.

D2: Approachability. ProgramAlly needs to be approachable for non-experts. To this end, it includes a set of methods to create and iterate on programs through multiple modalities, and users can choose what fits their needs. ProgramAlly should aim to have as little technical jargon as possible and explain program parameters in natural terms.

D3: Accessibility. ProgramAlly needs to be VoiceOver and Braille display accessible for users, both while creating and running programs. ProgramAlly’s VoiceOver implementation groups related parameters together to provide context for each statement. Additionally, ProgramAlly provides visual context while running programs to help the user aim and know what is in frame.

ProgramAlly is built on a generalizable representation of visual filtering tasks. Here, we describe how that represntation was designed, how it is implemented, and how it is used to run visual filtering programs.

ProgramAlly’s first method for creating new filtering programs is a block-based programming interface, shown in Figure 1. This block mode can be used to create a program from scratch, or to edit a program that was generated automatically by one of the other two methods. When first creating a new program, the authoring interface will display a program with two empty items in order to provide a default structure for users to fill in. There are two sections of this interface. First, a heading called ‘Program Summary’, which includes a natural language summary of the implemented program for users to refer back to as they edit. For the default program, this will initially read, “Find any object on any object”, and will update as users fill in the program with their desired items.

Next, under a heading called ‘Edit Program Directly’, users can read through each statement in the program, and edit them with actions in VoiceOver. For example, when VoiceOver focus is on the first ‘find’ statement, it will announce: “Find any object, actions available: Edit adjective, Edit object, Delete this item.” If parameters for the item have already been selected, the VoiceOver description changes to reflect what has been chosen. For example, for the statement ‘find RED BUS’, the description would be: “Find red bus, actions available: Edit adjective ‘red’, Edit object ‘bus’, Delete this item.” Grouping these together as one single element with multiple actions, rather than having ‘edit adjective’ and ‘edit object’ as separate VoiceOver elements, is meant to clarify that the different parameters in the ‘find’ statement are functionally related, without relying on the visual aspect of them each being on one line (D3: Accessibility). This design was also based on the VoiceOver experience of Apple’s Shortcuts app (Apple, 2022), where each block is read as a separate element and editing parameters can be similarly accessed through actions. When either of the edit actions are activated, a new page will appear listing the possible adjectives or objects to fill in the program (shown in Figure 1). The menu includes buttons that can be used to filter the items by type, or a search bar for finding a specific item.

Inspired by natural language programming approaches, ProgramAlly includes ‘Question Mode’, which generates a program from a question or statement (D2: Approachability). Users can type or dictate a question, and the resulting program will appear in the block-based interface for them to review and refine further. For instance, the query, ‘What does this bottle say?’ would result in the generated program: ‘find ANY TEXT on BOTTLE’. This result could then be modified with a follow up question: ‘Actually, just read the biggest text’ changes the program to ‘find LARGEST TEXT on BOTTLE’.

To prototype this interaction, we use a few-shot prompting approach with GPT-4. We provide a custom system prompt describing how to extract items, and listing the possible item classes. Then, we provide a set of approximately 20 queries and their correct JSON program representation that we wrote based on examples from accessibility datasets (Herskovitz et al., 2023; Gurari et al., 2018). Without developing a custom entity extraction workflow, we found that this approach works well. However, GPT-4 will sometimes produce errors. The most common issue is the model hallucinating new object classes. In this case, the block interface will alert the user that there is an unsupported field and open the editing menu for users to select an alternative. The model very rarely produces programs with an incorrect structure. If the model fails to extract entities, which can happen if the question is vague (e.g., “What is this?”) it will occasionally respond with natural language rather than a program (e.g., “I’m sorry, I don’t know what you mean, can you clarify?”). Future work could use additional fine-tuning to create a conversational approach to clarifying ambiguous language.

ProgramAlly also includes a method for users to edit programs with a follow-up question, rather than by manually editing a generated program using the block-mode. When editing a generated or pre-existing program, there is a text box where users can type or dictate a follow-up question (Figure 1). Various follow up questions are included in our system prompt to GPT. Based on feedback in our formative studies, we also included an option to access this feature while a program is running. If the program output is not as expected, users can directly access the option to edit the program with natural language, for rapid iteration. For example, when running the program ‘find NUMBER on BUS’, the user could provide the statement, “Read the route name instead”, and the program would be modified to be ‘find TEXT on BUS’.

ProgramAlly’s Explore Mode allows users to automatically generate a program by selecting a target feature detected in the camera feed (D2: Approachability). Explore mode lists all object and text features in the image, and users select an item to filter for, effectively providing a demonstration of the filtering behavior. Explore mode was included to address the challenge of unknown-unknowns (Bigham et al., 2017): without knowing what visual features or information is present, blind users would not necessarily have all of the information available to write a working filtering program.

In this mode, ProgramAlly runs all object and text detection models at once, with the goal of outputting everything that a user might want to create a program to find. Users then select the information they are looking for to demonstrate filtering behavior, and a program is generated which aims to filter for that type of information in the future. For example, as shown in Figure 3, the camera is pointing at a bus stop. If the user selects ‘30’, which is a route number, the resulting generated program will be ‘find NUMBER on BUS’, because that is where the text ‘30’ was found in the frame. Once a program is generated, the app again displays the result in the block-based interface, with the program summary and the option to edit the generated program further either with a question or with blocks.

Participants were recruited using email lists for local accessibility organizations, prior contacts, and snowball sampling. Participants were required to be over 18 years old, have some level of visual impairment, and regularly use a screen reader to access their devices. Participants were also required to have an iPhone so that they could download ProgramAlly via TestFlight.

Demographic information for participants is shown in Table 1. Of the 12 participants, two had some prior programming experience for their coursework or career. However, participants had a range of experiences with technology and VoiceOver, and not all were experts. For example, R2 and R4 were assistive technology professionals, while F2 was new to using VoiceOver and had not previously used any mobile assistive applications. We recognize that recruiting remote participants can create a bias for people who are technically savvy, as they need to have a desktop and be familiar with Zoom. To try diversifying our sample, we also recruited in-person participants.

After a brief introductory interview, participants were introduced to ProgramAlly by reading through a pre-written program to familiarize themselves with the concept and interface. Participants were then asked to modify the example program slightly by adding an adjective. We tried to keep verbal instructions minimal to let participants reason for themselves about the program. The only pointer that we gave to participants was that they could swipe up or down on the program elements to hear the different editing actions available, because depending on the verbosity settings of their device, VoiceOver may not have spoken this information.

After this introduction, participants then used ProgramAlly to create and run three programs. Participants used each of the three programming interfaces (block mode, explore mode, question mode), and were assigned tasks to create a program for (i.e., ‘create a program that will find addresses on mail’). While creating each program, participants were prompted to think aloud to explain their thoughts or concerns, or to ask questions. Participants then ran the programs they created, with remote participants using sample images and in-person participants using props. These props included books, grocery items, and packages and mail, and examples are shown in Figure 4. In-person participants were also given the option to compare their program to either Be My AI or Seeing AI, depending on what they would normally use for the same task. After creating each program, participants were asked to rate the ease of creating the program, and how accurate they felt the program was.

Finally, participants were asked to rank the three creation modes on different factors. The study concluded with an open-ended interview about the prototype app and the experience of using end-user programming methods for DIY assistive technology overall. Participants were asked to imagine how such an app could fit into their existing assistive technology workflows, and the pros and cons of creating programs to customize assistive technology.

Remote participants joined a Zoom call that was recorded, and when testing the app they were asked to share their phone’s screen in the call. Similarly, the face-to-face participants interviews were audio recorded, and their devices were screen recorded. The audio was later transcribed and used for analysis. We then created written descriptions of participant’s strategies for completing each tasks from the video data.

Since participants were encouraged to use a think-aloud method and take their time to fully explore the functions, ask usability questions, and give feedback, performing an analysis of task completion time does not provide much insight about how ProgramAlly works in practice. Instead, we primarily report qualitative data on participants’ strategies and workflows while using ProgramAlly, and their general feedback.

Participants generally felt positively about using filtering programs in ProgramAlly. As P1 described: “There is an abundance of visual information. And sometimes a blind person is short on time, and you really just want the particular piece of information you’re curious about” (P1). Not all of the example filtering programs were equally useful to participants. For example, R1 appreciated the ‘find PERSON on BENCH’ program and envisioned using it in a local park with many benches, while R3 had a seeing eye dog already trained to do this task. All participants saw practical use in at least one of the filters they tried, and all were able to come up with ideas for filters they could create in the future.

Most participants felt like with practice, they could become more familiar with the system and would be able to quickly create new programs. However, as blind end-user developers they also faced unique programming challenges.

ProgramAlly’s block-based programming interface has the advantage of offering more fine-grained control over a program, but it also has the highest learning cost. However, participants expressed that this was something they felt confident in being able to learn over time as they created new filters.

Participants considered question mode to be the fastest way to get started making programs. However, the generated programs sometimes did not capture the correct parameters.

Participant appreciated the potential of explore mode to give them a new awareness of visual features. But, in its current state, it was difficult for participants to know what features to select, and the generated programs sometimes contained unexpected conditions.

Participants generally appreciated all three programming modes were available in the app. An overview of how participants compared the modes is shown in Figure 5. Each mode required participants to think about their goal in a new way, and required different types of effort to turn it into an operable program. Parameterizing a request in block mode, selecting a visual feature in explore mode, and phrasing requests in question mode each presented their own strengths and challenges. Because of this, participants could see benefits of using different programming interfaces in different scenarios. As R4 put it: “It’s all contextual, I think. So it depends on what you want to do. Like, we did 3 different examples. But I would use different methods based on what I knew about the environment. It just depends on what you’re doing, you know, if you already have an image you’re working with. You might go with that particular program, you know, you explore the image, and then that’s what you use. It just really depends on the situation” (R4).

Participants also noted that the different modes could be helpful to people with different levels of technical expertise. P3 said, “I think, also, like, to the user, they’ll be scared, like, I have to do programming to use the app… But I mean, anyone pretty much can do it, and there are multiple ways, even if one method is gone. So there were two other methods to create” (P3).

Participants appreciated the deeper level of customization available when programming filters as compared to current assistive technology. As R2 described, it puts power into the hands of the user to decide what they needed: “I think it all comes down to providing choice. Ultimately, what I like is that you’re putting the information available, in the person’s hands to choose… You know, just being able to empower people to be independent… What you’ve all created here is really neat because it’s creating modularity to access the information. And I love that. I love that. And I wish more and more assistive technology companies thought about, how can we take these pieces of information and put it in the hands of the people that need it in a way that they can then modify it and change it and make it their own” (R2).

On the other hand, R1 noted that having to put in the effort to create filters themselves could be considered a burden: “People are not developers. Another developer, I suppose, knows how to program… I’m not a builder. There are tasks that are difficult for us to do, but what, I have to spend 5 hours to tinker with a program for what?” (R1). R1 expressed that they felt like ProgramAlly was a good option for developers wanting to quickly create things, but that it may not be ideally directed towards end-users. Other participants who did not mind the idea of programming still mentioned that re-framing the functionality might make ProgramAlly feel more approachable. Eventually, ProgramAlly could be considered as a platform for people to share programs that they have created, enabling a level of collaboration among end-users with different levels of expertise.

Throughout our user studies and analysis of existing data, we encountered many scenarios that could be addressed by ProgramAlly with the addition of new program operators. While ProgramAlly was implemented with ‘find’ and ‘on’ statements for simplicity and approachability, the addition of new operators could make ProgramAlly more expressive for DIY enthusiasts and power users. For example, the addition of traditional logical operators such as AND, OR, and NOT would allow for a greater degree of specificity in programs. However, AND and OR are easily confused among end-users as their natural language counterparts can be ambiguous, so introducing these would need to be done with care.

New operators could also define additional ways for objects to interact with each other. Currently, ‘on’ denotes objects whose bounding boxes are primarily overlapping. An operator like ‘nearby’ could specify items that do not overlap, but are in proximity. Similarly, ‘following’ could specifically find text content after a phrase, as in find TEXT following "EXP:" for more specifically finding an expiration date. ProgramAlly as a system could be extended with these, but it would require new block interface designs, a challenge for approachability and accessibility.

Additionally, ProgramAlly could benefit from the inclusion of additional, specialized models for certain tasks. For instance, a model that detects the make and model of a vehicle for locating a ride share, or text classification models that can filter out ‘brand names’ or ‘flavors’ for shopping scenarios. One notable object class missing in ProgramAlly is ‘digital display’, for reading screens on thermostats, microwaves, or buses. We attempted to use YOLO-World to detect this class, but found that it was not accurate enough to be usable. Although we currently use customized YOLO-World models on-device so the classes are pre-selected, YOLO-World was built for the ability to add new classes in real-time. In the future, when question mode extracts parameters from a request, the server could automatically generate new YOLO-World models to fill in gaps in the program as needed.

In the future, we imagine ProgramAlly being paired with other personalization approaches to create a deeper level of customization. For instance, if combined with the capabilities of teachable object recognizers, users would not only be able to locate personal items, but to integrate them into programs as a basis for further filtering or automation. Overall, we also see ProgramAlly as going beyond programming for visual information tasks. We believe our study reveals important findings about how blind end-users program, ideally leading to supporting more complex tools for people to DIY a range of assistive technologies outside of filtering programs.

Furthermore, when the creation ceiling in ProgramAlly is raised with things like additional operators and models, or if end-user programming approaches are eventually used to create different types of assistive technologies, this comes with a trade-off. Balancing this complexity with ease of use is a critical concern for future accessibility work in this direction. For instance, although ProgramAlly could eventually include a large library of models and classes to detect, these could be activated selectively or as-needed for different users or situations, making the system less overwhelming. Eventually, ProgramAlly could also make suggestions for how to create or improve programs based on how a person uses the system over time.

Because of the long-tail problem, some of our participants saw managing a library of programs as unwieldy. For example, R1 said, “Would I have to program actions for all the objects in the world?” (R1). P1 expressed a similar sentiment: “When it comes to just the variety of information that anyone might be looking for, at any given point of time… I don’t know. It just feels like there are so many permutations and combinations here. So many ways in which humans may want to query information that trying to build an even remotely comprehensive list of the more common categories of information seems like an endeavor that’s really hard” (P1).

Being able to automate when programs are run could remove some of this burden. For example, a ‘find ADDRESS on PACKAGE’ program could be automatically started whenever a package enters the frame, on the assumption that the user is sorting mail. Or, like other mobile automations, programs could be tied to a location. For example, when a user arrives at a bus stop, the ‘find NUMBER on BUS’ program could start.

Although large vision language models (VLMs) are becoming more powerful, they may not be a panacea, and making them truly beneficial requires deep integration with the needs of blind people. Despite the advantages of providing fully automated, subjective descriptions, they also add new challenges for blind users to acquire information. While this is still an area of active research, Massiceti et al. found the CLIP-based models were up to 15% less accurate on images taken by blind people (Massiceti et al., 2023). Additionally, as hallucinations seem to happen more often when describing complex scenes (Liu et al., 2024) or when being asked a follow-up question (the model appears to second guess itself), they could potentially arise more in accessibility contexts.

Generally, this also calls back to the long lived direct manipulation vs interface agents debate (Shneiderman and Maes, 1997); although there is an effort cost to creating personalizations, there is also a cost when an intelligent system assumes someone’s needs and gets them wrong. Although they may appear at odds, we envision end-user programming as a supplement, not a replacement for large VLMs. We envision the two approaches complementing each other in the following ways:

Reducing hallucinations by breaking down problems. Programs can serve as a way to break down visual problems into smaller pieces, avoiding complex questions that might cause models to fail. Just as ProgramAlly crops each image frame to relevant object bounding boxes when running programs, a cropped version of an image could be passed to the VLM to query in a more constrained way. For example, programs could contain subjective adjectives like ‘clean’ or ‘matching’. A VLM could be queried for these items in a constrained way, the answer could be parsed and fed back into the program. VLMs still are not good at reading large chunks of printed text or reasoning about complex scenes, but if the input and output were constrained then they may produce better results.

Balancing ambiguity in language and improving explainability. As discussed in our study findings, language is ambiguous, and programs can help articulate specific intents. Additionally, programs can serve as explicit step-by-step instructions of what a system is doing to come up with a given answer. This could help users better understand the limits of different tools, to better understand and predict why they fail.

Making large VLMs ‘live’ and creating reusable queries. Current large VLMs take in a static image as input. Yet, as models become faster, running a query on a live camera feed will not be as simple as repeating the question on each frame. Because programs specify what users want to hear and when, they could be used to convert natural language responses into real-time feedback.