Voicify Your UI: Towards Android App Control with Voice Commands

To provide direct invocation for different app components, Voicify fetches the system files from locally installed applications that specifies the list of components provided by the app. Thus, all installed applications will be compatible with Voicify without any additional integration from the other app developers, resulting in higher application feature coverage. In addition, local retrieval will prevent application version mismatch when the locally installed version is not up-to-date with the released version from the developers, which was a reported issue that caused Google Assistant to malfunction (Xaif et al., 2022).

Our system uses explicit intents for the direct invocation process, requiring an activity name or a deep link to determine the target component. These data can be found inside the Manifest.xml file within the application source code. Therefore, Voicify locates and retrieves the Manifest.xml file from the on-device application’s directories, as shown in Fig. 3. The data inside application directories are encoded in Android-specific binary XML format, allocated into chunks with little-endian byte order by default (Application, 2022). Therefore, we introduced a binary decoder to convert the Manifest file for each installed application into UTF-8-character representation.

Voicify retrieves the intent filter for each exported component, which acts as the access point to open that component. Using the attributes from retrieved intent filters, we construct corresponding intents that can directly invoke that intent filter. In Fig. 3, we illustrate the representation of deep links (blue boxes) and exported components (red boxes) in the manifest file for the YouTube application. Since a deep link is a unique resource identifier, an intent that attaches a deep link can directly invoke an app component. The system retrieves it by concatenating the host, scheme and path prefix in the <data> tag. To open application components that are not integrated with deep links, we use the package name and the activity name to identify the destination component. Voicify utilizes the declared action name such as SEARCH and VIEW to improve the granularity of feature exploration. All extracted intents will be used for direct invocation process in Section 4.3.

In Voicify, UI semantic retrieval is an essential module that allows the system to identify the UI elements for interactions. Similar to Kalysch et al. (Kalysch et al., 2018), we utilized Android Accessibility Service API to gain a system-level semantic understanding of UI elements. Accessibility API provides information about on-screen elements via AccessibilityNodeInfo objects (Developers, 2022a), encapsulating the data of UI components.

Voicify retrieves interactive UI elements and classifies whether they are clickable, scrollable or editable. We use the extracted textual data from TextView to label interactive nodes, which are used for matching with the target from the users’ command. For non-TextView interactive elements, Voicify gains an understanding of them by investigating the effect of UI elements grouping (Zhang et al., 2021). Specifically, we obtain its label by searching for the adjacent TextView element which shares the same parent node, inspired by the label searching algorithm in Yang’s previous work (Yang et al., 2020). We also extend the search range to retrieve textual information from the child components which is located inside the interactive element to improve the algorithm’s coverage. Voicify uses spatial information about UI elements to identify adjacent and overlapping elements, which provides contextual information to improve the semantic analysis of the UI.

On several occasions, an interactive element (e.g., an icon) is not attached with any visible labels on the screen, hence the user is unable to specify the UI element by mentioning its label. Following the implementation of Voice Access (Hume, 2020), we provide a tooltip labelling system, where a distinct number is dynamically assigned as a temporary label for unlabeled elements. By using the Layout Inflater, we augment a numbering tooltip next to the unlabeled UI element by tracking its absolute coordinator on the screen. In this way, users can interact with unlabeled elements on the screen using the given number. For example, in Fig. 4, the top right icon is appended number 1 since it does not provide any visible label, in which the user can tap it by saying “tap number 1”. Voicify also caters for users with special visual requirements by allowing runtime customization of tooltips’ appearance, including the size, colour and opacity.

Currently, the NLU system in the other voice assistants adopts frame-based meaning representations composed of intents and slots (Gupta et al., 2018; Louvan and Magnini, 2020). However, such representation is known as having difficulties dealing with complex logic such as conjunction and negation in natural language (Cohen, 2020). To avoid the restrictions of frame-based representations, we propose a novel MR language, VoicifyLang. VoicifyLang defines several actions in the user commands, such as tapping, scrolling or entering text onto the UI elements. VoicifyLang also categorizes the action targets such as apps, components in the apps, the buttons on the screen, the input text and the directions of scrolling into different primitive types.

We defined the language in a way that it is flexible to deal with more complex logic operations in the human language. Therefore, the language is highly extensible to support more features in Voicify. The detailed syntax of our meaning representation is described by Abstract Syntax Description Language (Wang et al., 1997).

Each MR in VoicifyLang comprises a sequence of tokens. We propose a novel parser, namely VoicifyParser, to convert each user command into a sequence of MR tokens. The architecture of VoicifyParser is mainly inherited from BERT-LSTM (Xu et al., 2020a), a semantic parser which uses an attention-based Sequence-to-Sequence neural network (Bahdanau et al., 2015) as the backbone. VoicifyParser builds a copying module and a schema encoding module onto the Sequence-to-Sequence to solve our task-specific problems, as illustrated in Fig. 5B. Specifically, the encoder of the Sequence-to-Sequence is a pre-trained language model, BERT (Liu et al., 2019; Kenton and Toutanova, 2019) and the decoder is a Long-short Term Memory (Hochreiter and Schmidhuber, 1997). The vanilla Sequence-to-Sequence model only generates the tokens stored in a fixed vocabulary. Specifically, the vocabulary for VoicifyLang contains the tokens of actions, targets and some special tokens such as the delimiters and placeholders. The actions are pre-defined by VoicifyLang, while targets, including apps, components and buttons, are extracted using the Data Collection module defined in Section 4.1. However, if the user command inputs the text, the text in the generated MR should all be copied from the user command utterance, given the VoicifyLang definition. Thus, the VoicifyParser employs a copying mechanism (Gu et al., 2016) that allows the parser to copy the text from the user utterance. Another problem is that the buttons on each app page are usually dynamically generated, which means many buttons are out of vocabulary. To solve this problem, VoicifyParser adopts a schema encoding mechanism (Wang et al., 2020). When the input user command intends to tap a button, Voicify collects all button names on the screen in runtime and sends them to the VoicifyParser. The schema encoding enables the VoicifyParser to generate tokens of a button out of the extracted buttons instead of only from the vocabulary.

The training data includes the pairs of user utterances, their aligned VoicifyLang MRs, and optionally a list of button names used to simulate the scenario in which the user wants to tap a button on a screen. Since manual annotation of user utterances is cost-intensive and time-consuming, we adopted a widely-used semi-automatic data collection method for semantic parsing, namely the Overnight approach (Wang et al., 2015), as illustrated in Fig. 5A. First, we manually wrote a set of synchronous context-free grammar (SCFG) (Chiang, 2007) rules. Expanding the SCFG rules could generate pairs of semantically equivalent canonical utterances and MRs. In the generated dataset, each MR has only one aligned canonical utterance. However, in the real-world scenario, natural language riches in linguistic variations. User speakers may vary word choices and morphologies for the same actions and targets in the MR and the syntactic structures for the utterances with the same MR. Therefore, we paraphrased each canonical utterance into multiple utterances with the same semantic meaning such that each MR would have multiple aligned utterances. The studies in (Wang et al., 2015; Xu et al., 2020b; Shiri et al., 2022) validate that such paraphrases could significantly improve the performance of the semantic parser. We applied automatic paraphrasing methods to reduce the paraphrasing cost. We used the best-performing paraphrase method, a commercial online paraphrase service, Quillbot²²2https://quillbot.com/, as in (Shiri et al., 2022). For each user command whose intention is to tap a button, we randomly sampled a list of buttons from the pre-defined set of buttons as the candidates for tapping. The button names are collected from the Rico dataset (Deka et al., 2017), which contains button names extracted from 10k Android UI screenshots. Thus, the parser can learn to generate tokens of the out-of-vocabulary buttons. The parser trained on the dataset generated by Overnight is proven to be robust to the richness and lexical and morphological diversities in the natural language (Wang et al., 2015).

The matching between application components and user-requested component consists of two steps. First, the system searches for the application name from the meaning representation in the installed application namespace. After identifying the matched application, Voicify searches for the component using the feature name and sends it to the executor. We search for the application name first to shrink the component search space and improve efficiency.

Since UI elements are dynamically changing when users are interacting with the phone, Voicify must continuously capture and analyse new UI elements. Voicify subscribes to the typeWindowContentChanged accessibility events (Developers, 2022b) to get notified when changes happen to user’s screen. The tool maintains separated dictionaries of labelled and unlabelled on-screen elements, which are updated using the data in AccessibilityNodeInfo objects delivered by Android OS, as described in Section 4.1. Upon receiving meaning representation, the matching module uses the attached UI element to get the corresponding node object, and deliver it to the executor to perform the action.

Android Accessibility API provides a wide range of system control methods that allows accessibility service developer to have full control over users’ device. Thus, Voicify used the performAction() method (Developers, 2022a) to perform actions, such as tapping, scrolling or entering text onto the UI elements. The executor sends the formed intent messaging object to directly open in-app components. On occasion, when none of the collected intents matches with the user’s request, we open the application entry screen as a baseline.

We implemented a queue containing action-target pairs to manage the sequential order of input actions. The executor will automatically pop the next action from the queue once the preceding action is successfully invoked, which allows command chaining to improve the usage efficiency. We implemented an automated validation system that verifies if the action is executed by monitoring the changes in the UI node metadata. Lastly, Voicify sends audio feedback to users as a confirmation that the system is ready for the next command.

We evaluate the parser with respect to its ability to convert the user command into the correct meaning representations. The parser is firstly trained on a synthetic generated dataset curated by the Overnight approach. Then the parser converts the commands in a manually collected test set into MRs. Then we compare the parser-generated MRs with the ground truth MRs and calculate the parser performance in three metrics.

For the test set, we first crawled user commands from the AndroidHowTo dataset (Li et al., 2020a), which contains step-by-step commands on achieving different tasks in Android. Then, we filtered out irrelevant data entries such as commands to interact with the other operating systems to obtain 101 clear commands for the test set. Each test case is a single command to perform the step, such as “Tap Clear browsing data” or “Go to the Profiles tab”. The parser requires a list of current on-screen elements for each tapping interaction. Therefore, we extracted the list of clickable elements for each tapping command from the Rico dataset (Deka et al., 2017), which contains the metadata to identify clickable elements in each screen from 10k Android apps. Finally, the ground truth MR of each test case is manually annotated by one student and validated by another to ensure the annotation correctness. Such a test set includes rich linguistic variations. For example, given our manual calculation, one MR has an average of 1.2 aligned natural language commands. The actions such as OPEN, PRESS, SWIPE and ENTER are described by 4, 6, 2, and 2 different phrases, respectively. Evaluating our parser on this test set can validate our parser’s robustness to the rich linguistic variations.

To evaluate the performance of the parsers, we adopt three evaluation metrics, Exact Match Accuracy (EM Accuracy) (Dong and Lapata, 2018; Li et al., 2021), Target F1 and Action F1. Exact Match Accuracy is one of the most commonly used metrics in semantic parsing, which computes the percentage of the commands which are correctly mapped to its ground truth meaning representations. To further understand how well the parser could predict the correct actions and targets, we also computed the F1 scores of the targets and actions. The Target F1 and Action F1 report the average F1 score of all targets and actions, respectively, weighted by their corresponding frequencies in the ground truths.

We compared with two baselines, vanilla Seq2Seq (Bahdanau et al., 2015) and BERT-LSTM (Xu et al., 2020b). Vanilla Seq2Seq includes an LSTM encoder and an LSTM decoder. BERT-LSTM replaces the LSTM encoder of Seq2Seq with a pre-trained language model, BERT (Kenton and Toutanova, 2019), and an additional copying module on the LSTM decoder. As our model is specifically designed to solve the issues of the out-of-vocabulary on-screen buttons, we also evaluated the performance of BERT-LSTM and VoicifyParser when all the buttons are not stored in the pre-defined vocabulary list. As such, the parser can only access the button names from the set of extracted on-screen buttons in runtime.

As in Table  1, our parser, VoicifyParser, can achieve an EM accuracy as high as 90% on the test set with rich linguistic variations. VoicifyParser also consistently outperforms the other two baselines in generating complete MRs or the individual actions and targets in the MRs. Compared with the vanilla Seq2Seq, the accuracy of VoicifyParser is even 40% higher. Due to a lack of a copying mechanism, Seq2Seq can not correctly parse most of the input-text commands. For example, enter ‘4-digit PIN code’ . is incorrectly parsed into ( ENTER , ‘ 4-digit PIN ’ ). In our experiment, it fails on all such types of questions in the test set, which occupies 5% of the total test set. Vanilla Seq2Seq employs no pre-trained language model, so it is not robust to the rich linguistic variations in the user commands. As a result, it performs significantly worse than BERT-LSTM and VoicifyParser in all three aspects. BERT-LSTM and VoicifyParser have comparable performance in terms of Action F1. However, BERT-LSTM lacks a schema encoding mechanism as our model. Thus, it can not utilize the runtime on-screen information. Even with all the buttons stored in the pre-defined vocabulary list, the Target F1 of BERT-LSTM is still 7% lower than VoicifyParser, depicting that on-screen information is a great boost to the parser’s ability with respect to predicting the targets if the parser employs the schema encoding. With all the button elements removed from the vocabulary, the performance of BERT-LSTM drops significantly. BERT-LSTM (OOV) performs even worse than the Seq2Seq parser in all three metrics. Surprisingly, removing the button targets also drops the action prediction performance for the BERT-LSTM (OOV). For our VoicifyParser (OOV), the influence of removing button names is negligible, showing that schema encoding improves the robustness of our model to the out-of-vocabulary buttons in the test commands.

In this experiment, we validate the ability of Voicify to directly open the installed application’s features.We consider an in-app feature as a specific screen that serves the functionality to users. Given a set of features from various installed apps, we measure the application feature coverage by manually opening each feature via voice command and recording the success rate. Since Voice Access does not support opening features from other apps using voice commands, we used Google Assistant (Assistant, 2022) as the baseline. Using the result from the formative study that investigated common user tasks on smartphone (Arsan et al., 2021), we identified the 10 most common categories that will be evaluated in this experiment. For each application category, we randomly picked 3 applications from the Google Play Store top suggestions. All chosen applications are popular, ranging from 1 million downloads to more than 1 billion downloads. For each application, we identify a set of provided features that the application developer stated in the Play Store listing description. The number of features from each app varies, ranging from 2 to 5 depending on the complexity of the app. For example, applications in Clock & Alarm category are simple, only allowing users to add new alarm and view the list of added alarms, while applications in Post a picture category like Instagram offers many more features. In the end, we collected 117 test cases from 30 apps, each of them being an important feature stated by the developer when publishing the app on Play Store. A full list of the apps and features can be found at the Github repository³³3https://github.com/vuminhduc796/Voicify.

For each test case, we first record the ground truth as the screen that can provide the requested functionality to users. We used the success rate as the primary metric, each test case will be marked as successful if the tool can directly open the desired screen, otherwise, it will be marked as failed. We compared the success rate of Voicify and Google Assistant to validate the performance of the direct invocation module.

Fig. 6 compares the feature coverage by category between Voicify and Google Assistant. Overall, Voicify successfully located and invoked 90 application features out of 119 test cases (76.9%), compared to 55 features from Google Assistant (47.0%). Both tools have achieved high performance in popular applications such as Yahoo Mail, SoundCloud and Twitter with over 80% success rate. However, the feature coverage from Google Assistant dropped dramatically for less well-known apps because those applications are missing the required declaration from their developers to integrate with Google Assistant. Since our approach does not require any developer efforts to integrate the application with Voicify, we achieved a solid performance across a wider range of apps.

We performed the error analysis by investigating failed test cases to understand the main problems that affect Voicify feature coverage. First, as mentioned before, some important features are encapsulated inside the application, which does not allow direct access from other applications. Hence, both Voicify and the baseline have poor performance for money transfer & banking apps as those applications might have enhanced protection, blocking direct access to certain features to prevent malicious attacks. Second, some application screens require data that is passed from the previous screen. This data is often stored in a bundle, attached to the intent to open. For example, when opening the video player from YouTube, the attached data must specify which video will be played using a certain video ID. Voicify is unable to identify and populate the required data, hence failing to directly open those features. Third, some applications have implemented a centralized activity as the entry point for all external invocations through intent. This implementation limited the list of features that is can be captured by Voicify, hence impacting the feature exploration of Voicify.

We performed a quantitative analysis of the time taken (in seconds) to complete each task, as shown in Fig. 7(A). In general, as the number of steps increase, participants required more time to complete the task. The average time taken to complete the tasks with Voicify is 93.2 seconds, compared to 140 seconds using Voice Access, resulting in a 33.4% efficiency boost. We observed a significant disparity in recorded time due to (i) the usage of direct feature invocation in task 1 with Voicify and (ii) the complexity in inputting text and tapping unlabelled icons with Voice Access in task 4. Specifically, participants directly opened the list of saved recipes in task 1 using Voicify, hence they skipped 3 steps as described in Fig. 7(B), resulting in a shorter time. For task 4, participants were requested to input text into several text boxes, which caused some issues in selecting the text box and typing the text. In addition, some steps in task 4 required users to tap an unlabelled icon (e.g., the star icon to add a note to the favourite list), therefore, users made several unsuccessful attempts to guess the corresponding label of the icon. Using the grid-based tapping from Voice Access was proven to be inefficient as it required extra commands from users to show/hide and change the granularity of the grid. Voicify solved the problems by attaching a numbering tooltip next to the unlabelled icon, allowing users to promptly perform interactions. By observing the experiment, we recognized the issue in transforming the human voice into the textual format. Voicify did better in post-processing the raw user command using predefined heuristics to resolve common transcription errors. However, we noticed that Voicify requires slightly more time to process each command because of the latency in API communication to the backend server. This latency caused a short period of unresponsiveness after recording the user command, adversely affecting the performance and user experiences. The problem could be mitigated in the future by deploying a mobile version of the model locally on user devices.

Fig. 8(A) summarizes the participant’s feedback on their level of cognitive load for each system using the NASA-TLX form. With a lower level of effort, participants achieved significant improvement in the performance using Voicify (t = -2.61, p = 0.035) as a result of direct invocation usages. In addition, the command parser was precise in mapping faulty commands to correct actions, helping users to easily interact with the system. Participants experienced less frustration using Voicify, as well as confirmed that Voicify required less mental demand, compared to Voice Access. The result proved a significant improvement of Voicify, fulfilling the design implication of reducing the required cognitive loads to operate the tool.

We compared the quantitative feedback of participants, including 10 design and usability questions on a 5-point Likert scale and applied a pairwise t-test to the result, as shown in Fig. 8(B). The result validated that we improved the usability of the voice controlling system, as the average SUS score for Voicify is 72.813, while Voice Access received 54.688. Voicify was better integrated than Voice Access (t = 2.65, p = 0.033). Due to the intuitiveness of the tooltip labelling system and direct invocation, Voicify received significantly better ratings across multiple indicators. The system is confirmed to be less cumbersome to use (t = -2.45, p = 0.041) and required less learning (t = -2.58, p = 0.036), compared to the baseline. Participants confirmed an improvement in the ease of operation (t = 1.99, p = 0.041) since tooltip selection was more convenient compared to the grid-based selection from Voice Access to tap unlabelled icons.

In this section, we collate qualitative feedback from participants after experimenting Voicify and Voice Access. Overall, the participants are satisfied with the tool, as well as providing suggestions for further improvements.

Innovative method of interacting with phone devices. Participants who have never used a voice control system to perform a sequence of daily life actions were eager to explore it further. P5 expressed that “I really enjoy using the voice to control the phone, it is a new concept to me". Moreover, participants who are tech savvy are impressed by the capacity of voice assistant technology, as evident in P6’s comment that Voicify “can do most of the things that I need" and “understand what I want to say". In normal circumstances, participants prefer the traditional tapping interaction over voice command despite their positive experience with Voicify. It is then concluded that even though voice command technology is excitingly novel and capable, the barrier to its wide adoption is the intricate human behaviour adaptation process.

Usages of direct invocation. Voicify ’s novel feature of direct invocation was implemented in the experiment and suggested as an option to carry out a sequence of activities, which garnered positive feedback from P1 and P7. Participants mentioned that the direct invocation feature was a great advance for voice control systems, as it allows users to “quickly access" the desired screen in a particular application, alleviating the complexity of the task by reducing the number of steps and wait time. Therefore, future improvement will include expanding the set of in-app components that can directly be opened from the main screen using Voicify.

Numbering tooltips as an effective solution. Voicify ’s tooltip labelling system received overall positive feedback. P2 mentioned that the best thing about Voicify was “the ability to identify icons in an app in numbers whereas in Voice Access one may need to use grid selections" and P4 appreciated the bespoke ability to tap on “labelling icons for which you may not know the name/label using numbers". P3, P5 and P7 agreed that numeric labelling has helped them to quickly interact with different icons on the screen without having to know the name of the icons, which is convenient and stress-free. In contrast, participants noted the demerit of the grid-based selection from Voice Access. P1 and P3 expressed that the grid-based selection is “difficult” and “imprecise”. The result showed the inefficacy of grid-based selection from Voice Access and the user preferences towards tooltips selection from Voicify.

UI design improvement & system feedback. Participants acknowledge the great UI design and responsiveness of Voice Access while giving Voicify suggestions to deliver better user experiences. P3 mentioned Voice Access’s “impressive capability to interpret the speech on the go” and P7 expressed great interest in the real-time “closed-caption” that Voice Access generated. For Voicify, P2 and P6 mentioned that the toggle button to start the tool and the system status indicator blocked certain UI components. To tackle this issue, possible solutions include moving overlaying components onto the notification bar or providing the ability to show or hide UI components of Voicify in runtime using a predefined voice command. P7 has suggested that displaying “closed-caption as we talk to give a sense of immediate feedback to the user", which will improve the responsiveness of the Voicify.