Security Apps under the Looking Glass:
An Empirical Analysis of Android Security Apps

Weixian Yao,1 Yexuan Li,1 Weiye Lin,1 Tianhui Hu,1 Imran Chowdhury,1 Rahat Masood,12 Suranga Seneviratne1
1 University of Sydney, Australia, 2 Data61-CSIRO,
Email: {first name.last name}@sydney.edu.au

Abstract

Third-party security apps are an integral part of the Android app ecosystem. Many users install them as an extra layer of protection for their devices. There are hundreds of such security apps, both free and paid in Google Play Store and some of them are downloaded millions of times. By installing security apps, the smartphone users place a significant amount of trust towards the security companies who developed these apps, because a fully functional mobile security app requires access to many smartphone resources such as the storage, text messages and email, browser history, and information about other installed applications. Often these resources contain highly sensitive personal information. As such, it is essential to understand the mobile security apps ecosystem to assess whether is it indeed beneficial to install them. To this end, in this paper, we present the first empirical study of Android security apps. We analyse 100 Android security apps from multiple aspects such as metadata, static analysis, and dynamic analysis and presents insights to their operations and behaviours. Our results show that 20% of the security apps we studied potentially resell the data they collect from smartphones to third parties; in some cases, even without the user consent. Also, our experiments show that around 50% of the security apps fail to identify malware installed on a smartphone.

I Introduction

Smartphones have become an essential element in our daily lives. As the end of 2019, the number of global smartphone users was predicted to reach 3.2 billion, which is a staggering 41% of the global population [44][43]. Due to the wide range of apps people use in their smartphones, smartphones have become a storage for highly sensitive personal data such as communications, images and videos, health information, and financial details. Also, many enterprises rely on smartphones for their business operations and in some cases even allow their employees to use their own devices for business work. Thus, smartphones have become lucrative targets for cyber attacks and it is of utmost important to keep smartphones secure.

Analogous to antivirus software for computers, mobile security apps are available for smartphones. Many of these mobile security apps are downloaded millions of times and many enterprises usually installs third party security apps or mobile device management software in company-issued phones as an extra layer of security [51]. While indicating that if the users stick to the official app stores and stay up-to-date with the operating system updates, they don’t need any mobile security software in their smartphones [41], both Apple and Google still allow third party security apps in their app stores. Whether the users need to install third-party mobile security apps in their smartphones is a contentious topic.

In multiple occasions some of mobile security apps have been reported for dubious activities such as intentional or unintentional personal information collection, advertisement fraud, or simply providing a false sense of security without actually detecting mobile malware [4][50]. The correct operation of a functional third-party-mobile security app requires a significant access to user data and mobile phone resources (i.e. in the from of permission requests). For example, a mobile malware detection app will require access to all the stored files to ensure that no malware is transferred to the device [3][2]. Similarly, a mobile internet security app will require access to all the URLs a user visits in order to prevent the user visiting a malicious website [7]. As a result, a smartphone user who is thinking of installing a mobile security app needs to make an informed decision whether it is beneficial to install such an app or simply stick to the security measures provided by the platform provider, i.e. majorly Google and Apple. To this end, in this paper we study the functionalities and inner workings of a set Android mobile security apps using metadata analysis, static code analysis, and dynamic behaviour analysis. Specifically, we make the following contributions.

•

We conduct a keyword search in Google Play Store and identify 328 Android security apps. We conduct an analysis of their metadata to understand the security app ecosystem. We show that despite increasing the security measures in the Android operating system, the number of security apps in the Google Play store is steadily increasing and they are frequently used by the end users. We find that 26 out of 328 apps were downloaded over 10M times and 37 apps were downloaded over 1M times.
•

We examine the privacy policies of 100 selected security apps and found that 55 of them may decide to share the data they collect with legal authorities or business affiliates. We also found that 20% resell the data to third parties, in some cases without user permission. Nearly, 70% of apps collect sensitive user information such as installed apps and device information.
•

Using static analysis we found that on average a security app requests approximately 22 permissions that include approximately four dangerous permissions. We observed that WRITE EXTERNAL STORAGE, READ EXTERNAL STORAGE, and READ PHONE STATE are the most frequently requested dangerous permissions. While these permission requests are essential for the correct functionality of the security app, they can as well be easily abused. At wrong hands, such level of information access together with the privacy consent users give during installation can have serious consequences.
•

We measure the effectiveness of the security apps in malware detection by copying and installing known malware samples into a test phone and checking whether security apps can detect them. We found that an alarming number of the apps fail to detect the presence of a malware. For example, only $\sim$ 20% of the apps were able to detect malware copies stored in a phone, and only $\sim$ 50% of the apps were able to detect malware installed in a phone. Also, we note that another significant fraction of security apps were not having up to date virus databases and as such fail to identify recent malware samples.
•

Finally, we analyse the network traffic generated by the security apps, and show that while majority of the traffic going out are encrypted, still there is unencrypted communication using port 80. We also analysed the type of information security apps sent to their remote servers and found that ‘device model’, ‘root build ID’, and ‘brand’ are the most frequently sent out information. Moreover, we also found evidence of sensitive personal information in the likes of installed apps and persistent identifiers (e.g. Android ID and BSSID) is being collected.

The rest of the paper is organised as follows. In Section II we describe related work and in Section III we describe our data collection methodology and present a basic characterization of the security apps. Privacy policy analysis and permission analysis are presented in Section IV and Section V respectively while Section VII delves into the network traffic generated by security apps. We discuss the implications of our findings, limitations, and concluding remarks in Section VIII.

II Related Work

There is a plethora of work studying characteristics and behaviours of mobile apps. We present related work under three topics; i) mobile malware, over-permissions and other dubious behaviours detection, ii) behaviour characterization of one specific group of apps, and iii) empirical studies of antivirus apps.

II-A Mobile malware, over-permissions, & dubious behaviors

When it comes to mobiles apps, malware detection is a topic that has been studied extensively [56, 11, 52, 35, 24, 46]. Early work by Zhou and Jian [56] was the first to study Android malware samples in the wild. Authors characterised over 1,000 malware samples belonging to 49 malware families, evolved during the early years of Android. Also, authors tested the detection rates of four mobile malware detection software and showed that in best cases 79.6% samples were detected and in the worst case only 20.2% samples were detected. Multiple subsequent work proposed methods to detect mobile malware. Such methods include static analysis [57, 15], dynamic analysis [53, 6], and hybrid methods [30, 42]. Tam et al. [49] and Faruki et al. [8] present comprehensive surveys of Android malware analysis techniques.

In contrast to mobile malware, apps requesting over-permissions do not pose a direct threat to the data stored in the phone or the device itself. Rather, these apps have more access to phone resources and data than what is required for the functionality of the app. Such additional access is usually used for advertising and analytics purposes and can be a threat to user privacy. Such access also allows possible future malicious activities. Grace et al. [12] and Leandros et al. [21] characterised the over-permission behaviours of top Android apps. Gorla et al. [10] clustered apps based on app descriptions and identified typical API usages for various app functionalities. Then the authors used anomaly detection to identify apps requesting unnecessary permissions. Other comparable work includes [27, 54, 26].

Another body of work focused on detecting various other malpractices in mobile app markets such as spamming [37, 38], counterfeiting [29], and ranking fraud [48, 58]. These apps adversely affect the overall health of the Android app ecosystem and in some cases were also reported to contain malware. Majority of the detection methods proposed use various features generated from app metadata like app description, developer information, ratings, and app icons in combination with machine learning models. In contrast to these works, our work specifically focus on the mobile security apps and study their operations and behaviours.

II-B Behaviour characterization of specific app groups

Several studies focused on specific functional groups of apps, especially from the privacy and security viewpoint. Meyer [25] et al. characterised the advertising behaviours of 135 apps targeting children under the age of five and showed that manipulative and deceptive behaviours are commonplace. Reyes et al. [33] conducted automated analysis of 5,855 most popular free children’s apps and found that the majority of the apps are in fact in violation of the Children’s Online Privacy Protection Act (COPPA). For example, 19% of the apps were found collecting personally identifiable information (PII) and 66% of the apps were still collecting the persistent device identifiers than the re-settable advertising identifiers. Similar studies has been conducted for health apps [22, 5]. More recently, He et al. [14] studied the behaviours of various dubious apps that emerged during the COVID-19 pandemic containing related keywords.

Ikram et al. [18][17] studied the privacy and security aspects of VPN apps and ad-blocking apps that are heavily used by privacy conscious smartphone users and the results were eye-opening. For instance, authors found that 18% of the VPN apps actually do not encrypt the traffic providing a false sense of confidentiality, and 4% of the apps contained malware. Similarly, Seneviratne et al. [36] and Hu et al. [16] showed that paid apps can also pose serious privacy and security threats to the users, despite the common belief that such problems predominantly occur in free apps. Our work is complementary to these studies. We provide an empirical analysis of a security apps; an app group that hasn’t received wider attention despite its high importance.

II-C Empirical studies of antivirus apps

Limited work studied of antivirus software in the context of desktop computers and laptops [45, 47, 28]. Rajab et al. [28] measured 15% of the all malware detected in the web were advertised as free antivirus software by analysing over 200 million malicious web pages. Stone-Gross et al. [45] delved into the economics of the scare-ware and fake antivirus market and showed that the organized underground business of fake antivirus software easily reaches the scale of hundreds of millions of dollars. Finally, Sukwong et al. [47] evaluated the effectiveness of the commercial antivirus software and found that they could not identify all the modern threats.

Our work is comparable to the industrial report [9] and the studies by Rastogi et al. [31] and Zheng et al. [55]. However, these studies solely focus on the effectiveness of the Android security software, whereas our study is multi-faceted. We not only study the effectiveness of the security software, but also investigate various other privacy and security threats they can pose.

III Data set and Basic Characteristics

In this section, we describe our data collection and pre-processing steps. We also, provide a basic characterisation of security app metadata.

III-A Data collection

We first conducted a keyword search in Google Play store with security related terms such as “antivirus”, “malware”, and “security” and identified 328 potential Android security apps. Next, we crawled the Play Store pages of these apps and downloaded the executable file (APK). We also recorded the metadata such as the app description, number of downloads, ratings, and developer information.

Since some of the analysis we subsequently conduct requires manual effort, we next select a representative sample of 100 apps out of the 328 apps we downloaded. First, we ranked the apps based on the number of downloads, number of user reviews, and rating, similar to what was done in previous work [37, 38]. Next, we divided 328 apps into three groups of 100, 100, and 128 apps, respectively. From the first group, we selected top-40 apps, and that included well known security apps such as McAfee and Kaspersky. We randomly selected 30 apps each from the other two groups. This selection gives us a sample of 100 apps that contains highly popular security apps as well as apps that are not so well known.

III-B Metadata Analysis

Timeline of Security Apps: To identify the trends in security app availability, we extracted the creation time from the metadata of each app. Figure 1 shows the development timeline of 328 antivirus apps as well as 100 apps that we selected. We notice a steady increase in the number of antivirus since 2010, and a steep increase since 2015. Overall, it indicates that there is an increasing demand for security apps attracting more developers.

Refer to caption — Figure 1: Timeline of security apps in Google Play Store

Geographical Location: We next analyse the geographical location of antivirus apps as illustrated in Figure 2. According to the figure, most of these apps are operating from the United States, Europe, China, and India. However there are some notable outliers. For example, well known Kaspersky security app is the one marked in Russia. One listed under New Zealand is “Emsisoft Mobile Security” - a security app that provides services such as malware scanning, anti-theft, privacy advisor, and web security. Similarly, apps in Indonesia and Philippines include Hackuna - Anti-Hack (a WiFi security app) and Search Phone Security - Booster & Cleaner,¹¹1This app is no longer available in Google Play Store respectively.

Popularity of Antivirus Apps: Figure 3 shows the download distribution the security apps. One app “Clean Master” was downloaded over one billion times. It is a multi-purpose security app providing variety of services such as app lock, battery saver, junk file cleaning, virus detection, and Android optimization. Nonetheless, this app was recently removed from Google Play Store under the suspicion that it is involved in an advertisement fraud scheme [34]. This indeed is a compelling incident to highlight how easily security apps can abuse the trust placed by their users.

“Device Care” by Samsung that provides device maintenance services for Samsung mobile devices was downloaded over 500 million times. The app analyses battery usage on a per-app basis and identifies battery draining apps. It also offers features such as power saving mode, remove unnecessary files, RAM management, malware detection, and device configuration. Potentially this app might be pre-installed in some Samsung devices, which can be a reason behind the higher number of downloads. Another app that had over 500 million downloads is “CM (Cleanmaster)”. This app was also from the same developer as the previous “Clean Master” app and was recently banned from Google Play Store. Finally, we note that there are no selected apps in the download range from one million to ten million because after the ranking we did (cf. Section III-A), from the first 100 apps we selected only the top-40 apps whereas from the other two groups we selected apps randomly.

Number of Downloads vs. Ratings: We next analyse the distribution of ratings (ranging from 0 to 5) against the number of downloads. As shown in Figure 4, the vast majority of the security apps have a very high rating (over 4), indicating they are well received by the end users. For example, all the apps with over 10 million downloads had a rating higher than 4.4. The banned “Clean Master” app had an average rating of 4.6.

III-C App description mining

Though all the apps we selected broadly fit into the category of Android security, they have different security functionalities such as “internet security”, “malware scanning”, or “virus removal”. To understand what types of services and functionalities the security apps provide, we next mine the app descriptions of the 100 apps we selected. Our approach is to use the doc2vec model [20] to obtain vector representations of app descriptions and cluster them using k-means clustering to identify functional groups.

doc2vec model - Since a large text corpus is required to train the doc2vec model, we start with 25,000 random app descriptions we downloaded from Google Play Store as a part of our previous work [29]. We pre-process the app descriptions with standard natural language processing techniques such as removing symbols, non-English characters, stop word removal, and stemming. Next, we train a doc2vec model using the resulting corpus that produces a vector representation of size 100 for a given text.

k-means clustering - We used k-means clustering to cluster the 100 selected apps based on the vector representations produced by doc2vec. We identified the optimal number of clusters based on the silhouette score, which is a standard metric used to measure the quality of clustering. The range of silhouette coefficient is [-1,1]. A higher value of silhouette score implies that there is high similarity between the elements of individual clusters and the clusters are well separated.

We calculate silhouette score for different $k$ values ranging from 2 to 10. The highest value of silhouette score resulted when $k$ was two. According to this clustering, Cluster-1 contained 65 apps whose major focus is on providing virus scanning functions and included keywords such as “scan”, and “protection”. Cluster-2 had 35 apps having keywords in the likes of “cleaner” and “boost”. These apps are mainly tools that facilitate removing unwanted files and cleaning up memory with extra features for security and virus detection. We highlight the main keywords in the two clusters in Figure 5.

IV Privacy Policy Analysis

TABLE I: Analysis summary of privacy policies

Type of data		Use of data		Data shared with third-parties
Attribute	No. of Apps	Attribute	No. of Apps	Attribute	No. of Apps
Personal information	71	Newsletter	53	Legal authorities	55
User behavior	62	Service improvement	77	Business affiliates	48
Hardware information	75	Security services	48	Business partners	55
Upload files to cloud	30			Re-sellers	21
				Others	30

As mentioned earlier in the introduction, smartphone users place an enormous amount of trust on the developers of security apps by entrusting them with their most personal data. Thus, it is important to understand whether there are any additional uses of data and resources accessible to security apps apart from the sole purpose of keeping user devices safe. Therefore, we next investigate the contents and the terms of the privacy policies of the 100 selected security apps. Surprisingly, five apps didn’t have any privacy policy. The most notable was “See Who Is Tracking You”, an app that allows the users to keep track of other apps collecting their location information in background. Currently over 500,000 users have installed this app without realising that there is no privacy policy.

We visited the ULRs of the privacy policies defined in the home pages of the apps and collected a copy of their privacy policies. Next, we read the privacy policies ourselves searching answers for three high level questions:

i)

What type of data the security apps collect? Since security apps have access to highly sensitive private data, it is important to understand whether they actually collect such data. Users have the right to know what data is collected and usually this information is hidden in the fine-print of the privacy policies. We could categorise the such data collections into four high level categories; personal information (e.g. personally identifiable information such as name, email, and persistent identifiers), user behaviour (e.g. how the app is used or other apps’ usage data), hardware information (e.g. device model and brand), and file upload (e.g. obtaining copies of stored data).
ii)

What are the intended uses of collected data? - Usually apps collect data for legitimate purposes such as measuring user experience, identifying bugs, or identifying feature requirements. While such data collections are understandable, many privacy regulations stipulate that app developers must clearly explain how the collected data is used. Thus, it is important to understand whether security apps comply to such legal requirements. We categorised the uses of data into three; newsletter (e.g. registering users into the developers mailing lists to receive information and promotional material), service improvements (e.g. identifying bugs), and security services (e.g. the core services provided by the app such as virus scanning)
iii)

Is the data being shared with third parties? If yes, who are the third parties with such access - In some occasions mobile app developers may share the data they collect with third parties. It is necessary for security app users to understand whether this is happening to their data and under what circumstances. It is equally important to understand who the above third parties are. For example, data can be shared offline in a business-to-business manner with analytics companies. Also, data can be shared online through embedded third-party advertisement and analytics libraries. Finally, data can be also shared on-demand with bodies such as law enforcement. We categorise such third parties into four; legal authorities (e.g. law enforcement requesting user data), business affiliates, business partners, and re-sellers (various business partners of the app developer). We also find occasions where the associated third party information is unclear. Such cases were categorised into other.

We adhered to manual reading than automated analysis because usually the privacy policies are complex legal documents with low readability scores [23] and as of now there is no automated tool that can be readily used. We tested the tool Polisis [13],²²2https://pribot.org/polisis nonetheless it was not able to conclusively provide reliable answers to the questions we were after.

In Table I we summarise our findings and the detailed analysis of each app policy with references to the paragraphs and the sections of privacy policies is available online.³³3https://tinyurl.com/ya8rhxgz We recommend interested readers to go through the table provided in the link for further information on app policies. We found that 55 apps may share the data with legal authorities if required and almost all the apps had some form of data sharing with their business partners. Around 70 apps were found to be collecting personal information and hardware information.

V Permission Analysis

We next analyse the permission usage of the selected apps. In Android operating system apps can only access phone resources be it hardware or data, only after declaring their intentions as permission requests in the Android manifest XML file associated with the app. According to Android documentation [1] there are three permission categories; normal permissions, dangerous permissions, and signature permissions. The normal permissions though they have to be declared by the app developer in the Android manifest file, do not require user’s explicit consent before accessing the data or the resource. Examples for normal permissions include access to the Internet, WiFi state, and detecting phone reboots. In contrast, dangerous permissions require user’s explicit consent before accessing the data and hence pop up a dialog window to request access when the app needs that permission for the very first time. Examples for dangerous permissions include access to contacts, storage, or location. Finally, signature permissions are a special type of permissions that is granted at installation time only if the app that attempts to use a permission is signed by the same certificate as the app that defines the permission. Signature permissions are usually used when a single developer developing a suite of apps.

The permissions requested by an app usually can give an understanding of the features of the app as well as any associated privacy and security risks. In order to check the permission requests, we decompiled each Android APK executable and extracted the AndroidManifest.XML. By parsing the Manifest file we extracted the permission requests defined by each app. We observed that, on average a security app requested 22.09 permissions. Out of that average number of normal permissions was 12.23, followed by 5.52 signature permissions and 4.33 dangerous permissions. ‘McAfee Security” requested the highest number of permissions by an app, which was 51. In Figure 6, we show the cumulative distribution of different apps. We notice that approximately 60% of the apps requested five dangerous permissions.

V-A Normal Permissions

We observe 149 different normal permissions were requested by the security apps. In Figure 7 we show the 10 most frequently requested permissions. RECEIVE_BOOT_COMPLETED is the most frequently sought permission which is understandable given that security apps needs to continuously run in the background across device reboots. Other frequently requested normal permissions are predominantly related to the internet access.

V-B Dangerous Permissions

We found that 35 different dangerous permissions were requested by the selected 100 security apps. Top-10 frequently requested permissions are shown in Figure 8. As can be seen from the figure, the two most frequently requested permissions were WRITE_EXTERNAL_STORAGE and READ_EXTERNAL_ STORAGE. The purpose of these two permissions is to read and write from external storage such as SD cards. More often than not, mobile users store personal information and data such as photos, documents, videos on external storage. Thus, despite the legitimate uses of these permissions in the likes of scanning for viruses or deleting malware and unused files, a bad actor can abuse this permission to steal personal information.

Similarly, another frequently asked permission READ_PHONE_STATE enables security features such as call blocking or blacklisting. However, granting this permission will also allow access to read the phone number or interrupt legitimate calls. In the privacy policy analysis we found that some applications collect the users’ phone number and they may use it for advertising and even share with third parties. Such information at wrong hands can cause major damages to the user such as identity theft.

To summarise, the majority of the requested dangerous permissions have an associated legitimate security feature included in the security app. However, in our privacy policy analysis we found evidence the collected data may have other uses and can be shared with third parties. As such, the security app users need to be cautious when granting such permissions and need to carefully evaluate the trade-offs between the feature and the potential privacy threats.

VI Malware Detection

TABLE II: The description of Six Selected Malware/Virus

Malware Name	Release Year	Makeup	Malicious Behavior
Trojan	2017	Fake Adobe Flash Player update the web page	1. Once installed, it pops up a dialogue box requesting a device information from a user.
			2. If a user grants access to device information then this malware contacts command and control (C&C) server and sends device information for possible exploitation.
Backdoor	2015	BeNews app (Already removed from the Google Play Store)	1. This malware successfully bypasses the security verification check from the Google Play Store and has been downloaded over fifty times by the device users before it is removed from the store.
			2. The malware works by opening the backdoor for hackers to remotely control the victim’s device and installing different types of malware(s) on a device, once a user clicks the app.
Spyware	2016	Banker Trojan (app name is various time to time)	1. Once installed on a mobile device, this malware hides its icon (in a stealth mode) and keeps sending the device information to the C&C server.
			2. This malware type also tricks user to enter or update their credit card details into a fake Google Play web page.
Banking malware	2016	Sberbank (a reputable Russian bank online app)	1. This malware attempts to perform a phishing attack on a user by creating an app logo that looks similar to the banking apps icon. Hence, it misleads a user to download the app and steals his credentials.
			2. The malware is also capable of intercepting SMS messages and incoming calls on a mobile device.
Banking malware (new)	2019	BatterySaverMobi	1. This malware was downloaded over one thousands times with dozens of fake five-star ratings at the Google Play Store.
			2. The malware gets a device administrator privileges by luring users to install a fake system update.
			3. It collects users banking information through inbuilt keylogger module and screenshots, and is much flexible in hiding itself than other similar Trojans.
Xbot	2016	Fake Google Plays payment interface	1. This malware imitates Google Play Store payment interface alongwith the login pages of seven banks to steal banking credentials and credit card information.
			2. The malware is also capable of control infected devices remotely.
			3. It also encrypts victim’s files and ask for the ransom to decrypt them.

We next evaluate how effective are these security apps in detecting known Android malware. We selected six malware samples that have been detected and publicly disclosed between 2015 and 2019. All sample malware are sourced from GitHub, and the general description of each selected malware is shown in Table II.

Using these malware we tested two scenarios; i) whether the security apps can detect a copy of malware stored in a phone and ii) whether the security apps can detect malware installed in the phone. We installed the selected security apps one by one in a Google Pixel phone and assessed its performance with respect to these two tasks. We highlight that we could test only 86 out of the 100 apps we selected because 14 of them were limited to Samsung phones only.

We show the results in Figure 9. Surprisingly, detection rate of disk copies were quite low. For each malware sample, only 15-20 of the security apps could detect them even after running a full system scan (when such a feature is available). Malicious file detection can be considered as the first line of defence when it comes to defending smartphones and lack of that indicates a serious functional limitation in commercial security apps.

The detection rates installed malware was high compared to disk copies, yet not perfect. Between 40-50 security apps out of 86, were able to identify installed malware except for the case of the banking malware. The detection rate of the banking malware for both on disk file and installed file was much lower than the others. The main reason for this may be the fact that this malware was released in 2019 whereas the others are a few years older. Nonetheless, it is an eye-opening result that 20-30 so called security apps fail to identify actual malware installed in a smartphone and also show much lower detection rates when it comes to more recent, yet publicly known malware. This gives a false sense of security to the users who are under the impression that their devices are safe due to the presence of a security app.

Further to this, as shown in Table III, 30 apps were not able to detect any of the installed malware and 12 detected only 1-3 malware samples. Only 32 were able to detect all six samples of installed malware. Example apps that did not identify app six malware samples includes “SandBlast Mobile Protect” and “Free Spyware Removal Info”, while apps such as “Norton”, “McAfee”, “Kaspersky”, and “Safe Security” were able to identify all six samples of the malware. We highlight that one factor behind lower detection rates is that there are some apps that provide different security features than virus and spyware scanning.

VII Network Traffic

Since the majority of the security app we investigated (95 out of 100), it is possible these apps might monetise user data through advertising and analytics. We already reported such evidence during our privacy policy analysis (cf. Section IV). To investigate further on this, we used the Lumen privacy monitor [32] which can continuously monitor all installed applications’ operations and record all traffic flows of each application separately. Lumen also can identify when personal information is leaked by a particular application.

To monitor all the traffic flows from each application, we installed applications one by one and set the monitor time to six hours. We granted all the permissions requested by the app so that it is fully functional. Next we extracted Lumen’s dataset for further analysis.

TABLE III: Detection of installed malware

No. of detected malware	No. of detecting security apps
0	30
1 - 3	12
4 - 5	11
6	32

VII-A Packet Destinations: IPs, Ports, and Domains

In Figure 10 we show the number of packets send by each app during the six hour observation period to different destination ports (top-10 according to number of packets as well as all apps). 360 Security is the app that sent out the highest number of packets. Majority of the apps sent encrypted packets to port 443 with the exception of APUS Security, which had a significant fraction of non-encrypted traffic going into port 80. Other apps with noticeable unencrypted traffic included 360 Security, Security Master, Virus Cleaner, and Super Cleaner.

Next, in Figure 11 we show the top-10 apps according to the number of unique IP addresses they contact. 360 Security was again at the top with contacting over 180 unique IP addresses accounting for over 100 domains. In summary, there were nine apps that contacted over 100 IPs and three apps that contacted over 50 domain names.

Most frequent domains that are being contacted by 360 Security are visualised in Figure 12. While the majority of the traffic has gone to cloud service providers such as Amazon, Google, DigitalOcean, and Rackspace, we also see the advertising company Mopub’s domain name among the domains indicating possible advertising or analytics activities.

VII-B Personal Information Leakage

Finally, we investigate what type of personal information are being collected by the security apps according to the Lumen report. Figure 13 summarises the findings. The top three frequently collected information are device model, build id, and device brand. Such types of information are potentially collected for record keeping (e.g. a virus was found only in certain brands of phones). Nonetheless, we also found a limited number of apps that collect more personal information in nature such as list of installed apps and personally identifiable information such as the Android serial, Android ID, and BSSID. While such information collection can still have legitimate uses in the context of device security, leaving such information in the hands of third-parties can pose significant privacy threats to the users [39][40].

All in all, it is surprising to see multiple security apps still not using fully encrypted connections. Also, it is questionable whether all of these network connections are essential for the core features of the app. Finally, we found evidence of security apps indeed collecting and transmitting personal information to their back-end servers and users need to be wary of this, given our findings on privacy policies.

VIII Discussion and Concluding Remarks

In this paper we presented the first multi-faceted study of Android security apps. We next discuss the implications of our findings, limitations, and possible extensions.

Providing a false sense of security - One of our key findings was that a significant fraction of Android security apps provide a false sense of security. Around 75% of the apps failed to identify malware copied into the phone storage. While the detection rates were better when it comes to installed malware ( $\sim$ 50%), they were not ideal. Also, we noted that detection rates drops further when it comes to very recent malware. Therefore, the users must not by any means consider their devices secure due to the mere fact that they have installed an Android security app. They must read the product reviews and understand the features provided by these apps. Most importantly, users must understand apps’ limitations as well as the features not provided by these apps.

Not following best software development practices - We also found evidence of developers not following best software development practices. For example, multiple apps were generating significant volumes of unencrypted traffic and there were a few apps without privacy policies. Also, as highlighted earlier some virus databases are not updated frequently. While both Google and Apple takes active measures [34][19] to control fraudulent security apps, it appears that much stricter checks are required.

Personal information collection and usage - Through permission analysis we demonstrated that the security apps have access to vast amount of personal data. Our privacy policy analysis showed that the majority of the security apps do indeed collect personal information and we found empirical evidence of such data collections in network traces as well. The users must be wary of these data collections and must carefully assess the trade-offs between the security features they require and the possible personal data leakage. It is important to read the fine-print of the privacy policies and understand how the collected data is used. Finally, to assist the end users in making their decisions, we believe it is a good practice for the app developers to publish descriptions of the uses of various permission requests.

Limitations and future work - Our analysis can be further extended in multiple ways. For example, further analysis can be conducted to trace what data is going to what domains by using an SSL proxy. That will allow to further segregate and decide whether the data is collected for the core features of the app or for advertising purposes. Also, privacy policies can be studied together with actual data collections, along side with privacy law experts and investigate whether the apps comply the applicable privacy laws and regulations. Permission analysis can be further expanded by decompiling the codes and analyse the actual API calls that are being called. This will allow to obtain a much fine-granular understanding of apps behaviours. In this work, we investigated only the effectiveness of the security apps in detecting malware. However, among the apps we investigated, there were apps that did not claim to provide malware scanning features, but other security features. Thus, further analysis can be done to investigate the performance of the security apps with respect to their specific features.

References

[1] Android Developer Documentation, “Permissions overview,” https://developer.android.com/guide/topics/permissions/overview.
[2] Avast, “Permissions required by avast mobile security,” https://support.avast.com/en-us/article/Mobile-Security-permissions, 2020.
[3] Bitdefender, “Bitdefender mobile security for android: Frequently asked questions,” https://www.bitdefender.com/consumer/support/answer/19987/, 2020.
[4] Brian Barrett, “Most android antivirus apps are garbage,” https://www.wired.com/story/android-antivirus-apps-bad-fake/, 2019.
[5] T. Dehling, F. Gao, S. Schneider, and A. Sunyaev, “Exploring the far side of mobile health: information security and privacy of mobile health apps on ios and android,” JMIR mHealth and uHealth, 2015.
[6] W. Enck, P. Gilbert, S. Han, V. Tendulkar, B.-G. Chun, L. P. Cox, J. Jung, P. McDaniel, and A. N. Sheth, “Taintdroid: an information-flow tracking system for realtime privacy monitoring on smartphones,” ACM Transactions on Computer Systems (TOCS), 2014.
[7] F-Secure, “Safe for android app permissions,” https://www.f-secure.com/en/legal/permissions/safe-for-android, 2020.
[8] P. Faruki, A. Bharmal, V. Laxmi, V. Ganmoor, M. S. Gaur, M. Conti, and M. Rajarajan, “Android security: a survey of issues, malware penetration, and defenses,” IEEE communications surveys & tutorials, 2014.
[9] R. Fedler, J. Schütte, and M. Kulicke, “On the effectiveness of malware protection on android,” Fraunhofer AISEC, vol. 45, 2013.
[10] A. Gorla, I. Tavecchia, F. Gross, and A. Zeller, “Checking app behavior against app descriptions,” in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 1025–1035.
[11] M. Grace, Y. Zhou, Q. Zhang, S. Zou, and X. Jiang, “Riskranker: Scalable and accurate zero-day Android malware detection,” in Proceedings of the 10th international conference on Mobile systems, applications, and services, 2012, pp. 281–294.
[12] M. C. Grace, W. Zhou, X. Jiang, and A.-R. Sadeghi, “Unsafe exposure analysis of mobile in-app advertisements,” in Proceedings of the fifth ACM conference on Security and Privacy in Wireless and Mobile Networks, 2012, pp. 101–112.
[13] H. Harkous, K. Fawaz, R. Lebret, F. Schaub, K. G. Shin, and K. Aberer, “Polisis: Automated analysis and presentation of privacy policies using deep learning,” in 27th $\{$ USENIX $\}$ Security Symposium ( $\{$ USENIX $\}$ Security 18), 2018, pp. 531–548.
[14] R. He, H. Wang, P. Xia, L. Wang, Y. Li, L. Wu, Y. Zhou, X. Luo, Y. Guo, and G. Xu, “Beyond the virus: A first look at coronavirus-themed mobile malware,” arXiv preprint arXiv:2005.14619, 2020.
[15] J. Hoffmann, M. Ussath, T. Holz, and M. Spreitzenbarth, “Slicing droids: program slicing for smali code,” in Proceedings of the 28th Annual ACM Symposium on Applied Computing, 2013, pp. 1844–1851.
[16] Y. Hu, H. Wang, L. Li, Y. Guo, G. Xu, and R. He, “Want to earn a few extra bucks? a first look at money-making apps,” in 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2019, pp. 332–343.
[17] M. Ikram and M. A. Kaafar, “A first look at mobile ad-blocking apps,” in 2017 IEEE 16th International Symposium on Network Computing and Applications (NCA). IEEE, 2017, pp. 1–8.
[18] M. Ikram, N. Vallina-Rodriguez, S. Seneviratne, M. A. Kaafar, and V. Paxson, “An analysis of the privacy and security risks of android vpn permission-enabled apps,” in Proceedings of the 2016 Internet Measurement Conference, 2016, pp. 349–364.
[19] Karissa Bell, “’Virus-scanning’ iOS apps have always been a scam. now Apple is finally cracking down,” https://mashable.com/2017/09/15/apple-app-store-virus-scanning-apps/.
[20] Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in International conference on machine learning, 2014.
[21] I. Leontiadis, C. Efstratiou, M. Picone, and C. Mascolo, “Don’t kill my ads! balancing privacy in an ad-supported mobile application market,” in Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications, 2012, pp. 1–6.
[22] B. Martínez-Pérez, I. De La Torre-Díez, and M. López-Coronado, “Privacy and security in mobile health apps: a review and recommendations,” Journal of medical systems, vol. 39, no. 1, p. 181, 2015.
[23] A. M. Mcdonald, R. W. Reeder, P. G. Kelley, and L. F. Cranor, “A comparative study of online privacy policies and formats,” in International Symposium on Privacy Enhancing Technologies Symposium, 2009.
[24] N. McLaughlin, J. Martinez del Rincon, B. Kang, S. Yerima, P. Miller, S. Sezer, Y. Safaei, E. Trickel, Z. Zhao, A. Doupé et al., “Deep android malware detection,” in Proceedings of the Seventh ACM on Conference on Data and Application Security and Privacy, 2017, pp. 301–308.
[25] M. Meyer, V. Adkins, N. Yuan, H. M. Weeks, Y.-J. Chang, and J. Radesky, “Advertising in young children’s apps: A content analysis,” Journal of Developmental & Behavioral Pediatrics, 2019.
[26] R. Pandita, X. Xiao, W. Yang, W. Enck, and T. Xie, “ $\{$ WHYPER $\}$ : Towards automating risk assessment of mobile applications,” in Presented as part of the 22nd $\{$ USENIX $\}$ Security Symposium ( $\{$ USENIX $\}$ Security 13), 2013, pp. 527–542.
[27] H. Peng, C. Gates, B. Sarma, N. Li, Y. Qi, R. Potharaju, C. Nita-Rotaru, and I. Molloy, “Using probabilistic generative models for ranking risks of android apps,” in Proceedings of the 2012 ACM conference on Computer and communications security, 2012, pp. 241–252.
[28] M. A. Rajab, L. Ballard, P. Marvrommatis, N. Provos, and X. Zhao, “The nocebo effect on the web: an analysis of fake anti-virus distribution,” 2010.
[29] J. Rajasegaran, N. Karunanayake, A. Gunathillake, S. Seneviratne, and G. Jourjon, “A multi-modal neural embeddings approach for detecting mobile counterfeit apps,” in The World Wide Web Conference, 2019, pp. 3165–3171.
[30] S. Rasthofer, S. Arzt, M. Miltenberger, and E. Bodden, “Harvesting runtime values in android applications that feature anti-analysis techniques.” in NDSS, 2016.
[31] V. Rastogi, Y. Chen, and X. Jiang, “Droidchameleon: evaluating android anti-malware against transformation attacks,” in Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security, 2013, pp. 329–334.
[32] A. Razaghpanah, R. Nithyanand, N. Vallina-Rodriguez, S. Sundaresan, M. Allman, C. Kreibich, and P. Gill, “Apps, trackers, privacy, and regulators: A global study of the mobile tracking ecosystem,” 2018.
[33] I. Reyes, P. Wijesekera, J. Reardon, A. E. B. On, A. Razaghpanah, N. Vallina-Rodriguez, and S. Egelman, “Won’t somebody think of the children? examining COPPA compliance at scale,” Proceedings on Privacy Enhancing Technologies, vol. 2018, no. 3, pp. 63–83, 2018.
[34] Robert Williams, “Google removes 600 apps from google play to reduce ad fraud,” https://www.mobilemarketer.com/news/google-removes-600-apps-from-google-play-to-reduce-ad-fraud/572700/.
[35] J. Sahs and L. Khan, “A machine learning approach to Android malware detection,” in 2012 European Intelligence and Security Informatics Conference. IEEE, 2012, pp. 141–147.
[36] S. Seneviratne, H. Kolamunna, and A. Seneviratne, “A measurement study of tracking in paid mobile applications,” in Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks, 2015, pp. 1–6.
[37] S. Seneviratne, A. Seneviratne, M. A. Kaafar, A. Mahanti, and P. Mohapatra, “Early detection of spam mobile apps,” in Proceedings of the 24th International Conference on World Wide Web, 2015, pp. 949–959.
[38] ——, “Spam mobile apps: Characteristics, detection, and in the wild analysis,” ACM Transactions on the Web (TWEB), vol. 11, no. 1, pp. 1–29, 2017.
[39] S. Seneviratne, A. Seneviratne, P. Mohapatra, and A. Mahanti, “Predicting user traits from a snapshot of apps installed on a smartphone,” ACM SIGMOBILE Mobile Computing and Communications Review, vol. 18, no. 2, pp. 1–8, 2014.
[40] ——, “Your installed apps reveal your gender and more!” ACM SIGMOBILE Mobile Computing and Communications Review, vol. 18, no. 3, pp. 55–61, 2015.
[41] Simon Hill, “Android phones are safer than you think, says google’s head of android security,” https://www.digitaltrends.com/mobile/googles-adrian-ludwig-says-android-is-more-secure-than-ever/, 2017.
[42] M. Spreitzenbarth, F. Freiling, F. Echtler, T. Schreck, and J. Hoffmann, “Mobile-sandbox: having a deeper look into android applications,” in Proceedings of the 28th Annual ACM Symposium on Applied Computing, 2013, pp. 1808–1815.
[43] Statista Inc., “Global smartphone penetration rate as share of population from 2016 to 2020,” https://www.statista.com/statistics/203734/global-smartphone-penetration-per-capita-since-2005/, 2020.
[44] ——, “Number of smartphone users worldwide from 2016 to 2021,” https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/, 2020.
[45] B. Stone-Gross, R. Abman, R. A. Kemmerer, C. Kruegel, D. G. Steigerwald, and G. Vigna, “The underground economy of fake antivirus software,” in Economics of information security and privacy III. Springer, 2013, pp. 55–78.
[46] G. Suarez-Tangil, S. K. Dash, M. Ahmadi, J. Kinder, G. Giacinto, and L. Cavallaro, “Droidsieve: Fast and accurate classification of obfuscated android malware,” in Proceedings of the Seventh ACM on Conference on Data and Application Security and Privacy, 2017, pp. 309–320.
[47] O. Sukwong, H. Kim, and J. Hoe, “Commercial antivirus software effectiveness: an empirical study,” Computer, no. 3, pp. 63–70, 2010.
[48] D. Surian, S. Seneviratne, A. Seneviratne, and S. Chawla, “App miscategorization detection: A case study on google play,” IEEE Transactions on Knowledge and Data Engineering, 2017.
[49] K. Tam, A. Feizollah, N. B. Anuar, R. Salleh, and L. Cavallaro, “The evolution of Android malware and android analysis techniques,” ACM Computing Surveys (CSUR), vol. 49, no. 4, pp. 1–41, 2017.
[50] Trend Micro Inc., “Apps disguised as security tools bombard users with ads and track users’ location,” https://blog.trendmicro.com/trendlabs-security-intelligence/apps-disguised-security-tools-bombard-users-ads-track-users-location/.
[51] Verizon, “Mobile security index,” https://enterprise.verizon.com/en-au/resources/reports/mobile-security-index/, 2020.
[52] D.-J. Wu, C.-H. Mao, T.-E. Wei, H.-M. Lee, and K.-P. Wu, “Droidmat: Android malware detection through manifest and API calls tracing,” in 2012 Seventh Asia Joint Conference on Information Security, 2012.
[53] L. K. Yan and H. Yin, “Droidscope: Seamlessly reconstructing the $\{$ OS $\}$ and dalvik semantic views for dynamic android malware analysis,” in Presented as part of the 21st $\{$ USENIX $\}$ Security Symposium ( $\{$ USENIX $\}$ Security 12), 2012, pp. 569–584.
[54] Y. Zhang, M. Yang, B. Xu, Z. Yang, G. Gu, P. Ning, X. S. Wang, and B. Zang, “Vetting undesirable behaviors in android apps with permission use analysis,” in Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, 2013, pp. 611–622.
[55] M. Zheng, P. P. Lee, and J. C. Lui, “Adam: an automatic and extensible platform to stress test android anti-virus systems,” in International conference on detection of intrusions and malware, and vulnerability assessment. Springer, 2012, pp. 82–101.
[56] Y. Zhou and X. Jiang, “Dissecting android malware: Characterization and evolution,” in 2012 IEEE symposium on security and privacy, 2012.
[57] Y. Zhou, Z. Wang, W. Zhou, and X. Jiang, “Hey, you, get off of my market: detecting malicious apps in official and alternative android markets.” in NDSS, vol. 25, no. 4, 2012, pp. 50–52.
[58] H. Zhu, H. Xiong, Y. Ge, and E. Chen, “Discovery of ranking fraud for mobile apps,” IEEE Transactions on knowledge and data engineering, vol. 27, no. 1, pp. 74–87, 2014.

Security Apps under the Looking Glass: An Empirical Analysis of Android Security Apps