NiCro: Purely Vision-based, Non-intrusive Cross-Device and Cross-Platform GUI Testing

Due to the massive number and enormous diversity of modern mobile devices, a mobile app’s running environments, including software and hardware, would vary dramatically and unpredictably. This raises the strong demand for testing apps over multiple devices in the development process to tackle device fragmentation [1]. Although numerous works were proposed to address the compatibility issue due to device fragmentation [17, 18, 19], most of them focus on the sole (individual) platform (e.g., Android) and are from an aspect of functionality rather than GUI. Compared to back-end functionality, the GUI is more subject to display changes on different and heterogeneous devices, such as the screen size and resolution, because the widget’s layout, appearance and even visibility may change responsively.

As shown in Fig. 1, the GUIs on diverse devices are differentially rendered even for the same page of the same app. The difference exists even for devices that have the same operating platform (i.e., Device 1 to Device 5, and Device 6 to Device 8). For example, many widgets on Device 1 cannot be filled in the same screen on a smaller device, such as Device 6. And the GUI layout may change responsively to fit the screen size. For instance, the widgets at the bottom are arranged in rows of four on Device 5, while they are squeezed into three-widget-row on Device 1 and two-widget-row on other smaller devices. These variances raise more challenges in locating and matching the widget cross-devices. Moreover, the app running on different platforms usually has to adjust its implementation, which causes some distinctions of the GUIs, hence complicating the cross-platform GUI testing. For instance, although Device 2 and Device 8 share a similar screen size, their GUIs are not identical (e.g., the icons at the bottom bar). This discrepancy usually lies in the different GUI layouts and nuances of widget appearance, though the related functionalities are usually consistent. All of the features complicate cross-device and cross-platform GUI testing.

Non-intrusive technique, contrary to intrusive technique, does not hack into the underlying system or app to acquire data or perform action [14]. It does not require connection to the target system through certain APIs and hence is not as sensitive to the variants of the programming running environment as the intrusive approaches. This feature is desirable to testing apps running on a volatile platform, such as Android which has dozens of major versions and distributions, where the API is frequently updated or changed and requires the intrusive approach to be revised accordingly[20]. In addition, intrusive techniques cannot correctly acquire information from apps in many situations, such as a closed system where the app metadata is inaccessible or unanalyzable [12]. Even for open source systems, some app metadata is still unable or inaccurate to obtain, e.g., runtime view hierarchy and metadata.

In contrast, the non-intrusive approach is more generic and less demanding. It obtains information and interacts with the app purely through an external interface, which is analogous to human-app interaction in that we mainly use eyes to view the GUI and fingers to operate the device physically. The non-intrusive approaches resort to the computer vision process to “see” and “understand” GUIs [21, 11, 16], and some works apply robot arms to conduct the external interactions with devices, similar to human fingers [22, 23, 14]. For the non-intrusive approach, the capacity of visual intelligence to detect and analyze the widgets on the GUI image decisively affects the overall performance. However, most of the existing non-intrusive testing approaches rely on preliminary computer vision techniques that cannot accurately locate and recognize GUI widgets from the GUI image[15], let alone the much more complicated cross-device and cross-platform testing.

Fig. 2 summarizes NiCro’s overall architecture. It comprises three modules at the system level: (1) a virtual Device Farm; (2) a Robotic System, and (3) a Host Computer. The first two modules provide various devices that NiCro interacts with in a black-box manner, and the third module runs NiCro’s core techniques to collect and process GUI information.

The visual analytic workflow is demonstrated in Fig. 2’s Host Computer component and illustrated in Fig. 3.

NiCro utilizes UIED [16], a state-of-the-art GUI widget detector, with some revisions to detect widgets on the GUI image. UIED is designed with the consideration of GUI’s special visual characteristics. This enables to achieve higher accuracy in GUI widget detection than deep learning models that are built for natural objects[15] (e.g., Faster RCNN [26] and YOLO [27]), or the methods relying on manually-crafted visual features [28, 29]. UIED contains three steps:

GUI Text Detection: The original UIED uses EAST [30], a scene text detector, to extract the text widgets. However, the EAST is only able to locate the text area but cannot recognize the text content, while NiCro requires the text content (if any) to represent and match widgets. Thus we replace EAST with the widely used Google OCR tool [31], which is trained to recognize a wide range of text images to detect the location and content of text widgets.

Non-text Widgets Detection: UIED applies a pipeline of image processing algorithms, including gradient-based binarization, flood-filling connected component labelling [32] and shape recognition. The GUI-specific approach is proven to be effective [15], but it suffers from the problem of losing nested widgets (e.g., widgets in a container). NiCro tackles this problem by recognizing the container where all the nested components are kept. In particular, NiCro identifies a widget as a container if: (1) it is rectangular and (2) none of its inside objects are connected with the widget border. Besides, NiCro only categorizes the widgets as “Text” or “Non-text”as the detailed widget category is not involved in the later process.

Merge of Text and Non-text Widgets: This step not only integrates the detection results, but also cross-checks non-text widgets and filters out false-positive non-text widgets. UIED counts on the OCR’s text results and discards non-text widgets whose bounding box intersects with any text region.

Moreover, NiCro recognizes the screen region from the GUI photo taken by the Robotic System using heuristics: If (1) a widget’s height is larger than half of the photo’s height and (2) the widget contains multiple widgets and (3) the widget is not contained by any widget, it is a screen region.

The widget detection produces basic widget spatial and class information (i.e., size, location and text/non-text class). However, as stated previously, the GUI and the widgets on it may change in terms of placement and appearance on different devices and platforms. Consequently, it is necessary to extract more comprehensive multi-modal information to represent and match widgets. Figure 3 shows some examples of the widget information extracted by NiCro. Overall, the extractor obtains five attributes to represent a widget.

Spatial Location: The Spatial Location is acquired from the widget detection directly and includes (Top, Left) and (Bottom, Right) to indicate the coordinates of the top left and bottom right corners of a widget’s bounding box.

Shape: This attribute implies the widget’s geometric property. It uses a 4-tuple (Width, Height, Area, Aspect Ration) to measure the widget’s bounding box, which is calculated by the Spatial Location (i.e., ((Top, Left), (Bottom, Right))).

Image Clip: The extractor clips the widget’s image from the GUI image as its visual information using the Spatial Location. Not only does the non-text widget needs the image clip to show its visual content, but also the text widget uses the image clip to present its font properties.

Text Content: The Text Content is obtained from the widget detector’s OCR results directly, which can either be a single word or a line of texts. NiCro does not combine multiple lines into a paragraph to avoid incorrect merging and tricky parameter adjustment. So, a chunk of text that contains several lines would be identified as separate text widgets. Note that some non-text widgets also have text content if there is a piece of text within it (e.g., the text button).

Surrounding Widgets: NiCro also examines the widget from a contextual aspect. This information is conducive to identifying the widget on the Target Device, in particular when there are multiple widgets with the same appearance on the GUI. A widget’s parent and neighbours usually remain constant even when the GUI changes responsively to fit different devices. Thus, the extractor includes the parent widget (e.g., container) and the closest adjacent widgets in four directions (i.e., up, down, left and right) for a widget. Note that the parent could be None, and the number of the neighboring widgets is 0 to 4 depending on the specific widget.

NiCro utilizes the widget’s attributes extracted in the previous step to match its counterparts in other devices. These attributes capture the comprehensive information of the widget and impose different degrees of influence on the matching result. Therefore, different types (i.e., text and non-text) of widgets put distinct priorities upon these attributes while comparing. For the sake of presentation, we denote the Source Device GUI where NiCro records the action as $G_{s}$, the Target Device GUI where the action is replayed as $G_{t}$. We also denote the widgets in $G_{s}$ as $W_{s}$, and the ones in $G_{t}$ as $W_{t}$.

NiCro has to match the entire GUIs in two situations: (1) to determine if the page reaches the margin and (2) to check if the widget-independent action (i.e., swipe horizontally and scroll vertically) is complete on a different Target Device.

To investigate NiCro’s cross-device capability, our evaluation leverages 5 different Android devices, including physical and virtual ones, which have a wide range of screen sizes from 3.7” to 8.0”. For the cross-platform replay, our evaluation also includes two iPhone devices in different models and an iPad. Table I shows the details of the experimental devices and Fig. 1 demonstrates their GUIs. Due to the current Device Farm in NiCro does not support iOS, we only use physical iPhone devices that are interacted with the Robotic System.

Regarding the subject mobile apps, we select a total of 28 apps spanning 14 frequently-used categories (see Table III). Furthermore, we record and replay actions on the device’s home page. For example, tapping the icon to open an app and swiping & scrolling on the home page. The app selection considers the following three perspectives: (1) it covers a range of most popular app categories to show NiCro’s generalization capability; (2) the apps are widely downloaded so that they are stable to function in different devices and (3) the apps support both Android and iOS as to test NiCro’s cross-platform ability.

NiCro’s virtual Device Farm is implemented based on Android Studio in version 2021.1.1.23, which provides a variety of emulators with various Android versions. For the Robotic System, we leverage an UArm Swift Pro Robot Arm manufactured by Ufactory [35], a 4-axis robot that holds a stylus pen to mimic a human finger to interact with the touch screen. The camera used in our system is a 4K Logitech BRIO Webcam [36], and the surface is simply a black pad placed under the camera. Finally, the Host Computer we use in the experiment is a desktop and runs Ubuntu 18.04.

The evaluation first examines the matching accuracy for widgets and GUIs. We implemented a range of image matching methods and tested them on the same experimental data with NiCro. Different from many previous image-based non-intrusive testing systems that solely or primarily rely on the widget image [3, 14, 37], NiCro adopts multi-modal information to represent and match GUI widgets and the entire GUI. The evaluation includes the techniques applied in two closest non-intrusive testing works and two individual single-modal approaches NiCro utilizes. In particular, we evaluate (1) Template Matching used in RoScript; (2) SIFT used in LIRAT; (3) ResNet that NiCro applies to encode widget image and (4) Text Content Matching in NiCro to match text widget’s content and texts on some non-text widgets.

In the experimental process, we iteratively select a Source Device and use the rest of the other 7 devices as Target Devices. This yields a 8 x 8 matrix of results for each evaluated method, as shown in Fig. 5. From the collected 28 apps and the home page, we randomly chose 25 text widgets and 25 non-text widgets from each app. Overall, 1,450 (29 x (25 + 25)) widgets are used as the experimental subjects, and every widget is guaranteed to be visible on all of the 8 devices while being matched, although their relative position and size would vary in different devices. Besides, we randomly select 5 GUIs from each app on the 8 devices to examine the GUI matching, which brings a total of 1,160 (5 x 29 x 8) GUIs. Then, we apply the matching algorithms to find the most matched widget for a Target Widget and the most matched GUIs for a Target GUI on a Target Device. Finally, we manually inspect if the matching is correct and calculate the matching accuracy.

This part of the evaluation focuses on the system’s overall performance in replaying test cases and actions. For each app, we applied 3 to 4 test cases composed of 3 to 8 GUI actions (widget-dependent or widget-independent) in every test case. These test cases are all common usage scenarios that cover the general and specific functionalities of each app. For example, ”Change user name” for some apps that support user login, and ”Search and enroll the most popular Python course” for the Education app. This yields a total of 107 test cases containing 507 widget-dependent actions and 132 widget-independent actions. In the process of the experiment, we randomly select one device as the Source Device to record and then replay the actions on the rest of the 7 devices (Target Devices). Therefore, there are 639 (507 + 132) actions recorded and 4,473 (639 x 7) actions replayed. We here use a strict replay success criterion: An action replay is successful if it is accurately reproduced on all 7 Target Devices, and a test case is successfully replayed if all its actions are replayed correctly.

We observe three phenomena in these app usage scenarios. (1) Widget-independent actions (i.e., swipe or scroll) are commonly involved in the test cases, in which 66 (61.7%) of the 107 cases contain at least one widget-independent action when recorded. This is because it is very common that the screen size is not enough to fill all the page content, which requires scrolling or swiping to move the rest content to the screen scope. (2) The screen size variances across devices cause the recorded target widget on the Source Device not shown on smaller Target Devices. It happens to 43 test cases (40.2%) when they are replayed on some smaller devices, in which the screen needs to scroll to expose the target widget. (3) The cross-platform implementation inconsistency changes the GUI layout and widget appearance in some apps. 78 test cases (72.9%) experience inconsistency at various levels when replayed on different platforms and they may need more comprehensive information to match target widgets.

With the observation, we compare NiCro with the three latest and closest works: MAPIT[2], LIRAT[11] and RoScript [14]. Table II shows the comparison. It contrasts the three factors in the respective approaches (Cross-Platform; Cross-Device; Widget-Independent actions). We also present the ratio of available test cases for each work in the TC column. LIRAT and MAPIT do not support widget-independent actions, making them unavailable to handle 66 (61.7%) test cases. Moreover, the intrusive technique MAPIT requires the target widget’s metadata (e.g., accessibility id, text content etc.), while we find that MAPIT fails to obtain the information of 91 widgets (18%) out of the 507 widgets in the widget-dependent actions on one or more devices, because of the absence or redundancy of specific widget attributes in the metadata. Besides, the intrusive approach cannot acquire any content within an external component such as advertisement and WebView that is the primary element of some apps (e.g., web browsers). This makes additional 12 test cases fail to run on MAPIT, which leaves only at most 29 test cases (27.1%) available for it. Although RoScript’s approach solely considers the same UI on the same device, it supports both widget-dependent and widget-independent actions, and only requires the image as input, which enables it to run on all of the test cases in our experiment. And thus, we use RoScript as our baseline to examine NiCro’s overall performance.

Figure III summarizes the experimental results. Overall, NiCro achieves a promising performance for action replay, in which it accurately replays 86% widget-dependent actions and 85% widget-independent actions. In contrast, RoScript only succeeds in 40% and 23% of these actions. Regarding the test case replay, NiCro replays 63% of them without any manual correction (0-Correction), where the entire test case replay fails even if only one of its action replay fails. To investigate deeply, we examine the situation that allows one time of correction (1-Correction) for each test case and experiment NiCro and Rescript again, in which NiCro successfully replays 89% cases and significantly outperforms its baseline (20%).

Both of these two approaches’ performances vary in different categories of apps. In detail, NiCro performs better on apps containing more textual content than those containing more image elements. It achieves 0.93 and 0.91 widget-dependent replay accuracy on Email and Chat apps while only reaching 0.76 and 0.73 on Video and Photo apps. This is because it can better detect and match text widgets (as shown in Fig. 5), but for GUIs filled with complicated images or whose widgets overlap on images, its UIED-based widget detector cannot performs perfectly and hence causes cascading errors in widget matching. Besides, the replay accuracy on Finance apps (0.81) and Map apps (0.81) is slightly lower for a similar reason, where they contain many diagrams and have GUI widgets overlapping on these diagrams, which fails the widget detection. On the contrary, RoScrip’s widget-dependent replay is even worse in textual apps (0.34 on Email apps and 0.36 on Chat apps) than in apps with more images (0.43 on Photo apps), because its Template Matching-based image matching approach cannot handle text-widget matching. Overall, RoScript completes all widget-dependent action replay poorly also due to its incapable to properly work if the Target Widget is invisible in the Target Device due to the smaller screen, and is mostly incapable of matching widgets with different appearances across platforms. On the other hand, NiCro’s performance on widget-independent actions obviously surpasses RoScript, in which NiCro succeeds 112 cases out of 132 but RoScript only completes 30 cases. This poor performance of RoScript is because it replays scroll and swipe simply with the same distance in the recording phase. However, such practice is inappropriate if the screen sizes of the recording device and the replay devices are inconsistent. This confirms the effectiveness of NiCro’s widget-independent action replay approach based on the GUI matching.

(1) The apps involved could be a threat. To examine NiCro’s generalization ability and counteract this, we use as many as 28 popular apps in 14 common categories plus the home page in the experiment. However, there are more apps that would affect the experimental results, such as Game apps that are majorly composed of graphical elements, which the UIED-based detector would not handle perfectly [15]. (2) The tested devices could be another threat. Although we intentionally select a diverse range of devices in terms of physical or virtual presence, different screen sizes and operating systems, it may still miss some particular devices that affect the result. However, our evaluation confirms that NiCro can well perform the record and replay across physical and virtual devices with or without the same operating system. (3) The robotic system’s experimental environment could also be a threat. We run NiCro’s robotic system under a relatively stable environment with a steady light and fixed camera focus to acquire clear GUI photos. The quality of the GUI photo taken by the camera may change under different circumstances and affect the widget detection and matching. However, it is not difficult to adjust the physical setting to adapt to the new environment. (4) The code of LIRAT and RoScript are manually re-implemented by ourselves. In our communication with the authors of LIRAT and RoScript, they did not agree to release their source code and data for our comparative analysis purpose due to various restrictions, forcing us to re-implement their techniques. We tried our best and implemented them in a way that strictly followed the description in the original papers.