Authoring Platform for Mobile Citizen Science Apps with Client-side ML

Crowdsourcing is a distributed task assignment model where large groups of paid or unpaid participants submit their works, typically some form of data, using the internet, social media, and smartphone apps. Citizen science is a special type of crowdsourcing, where the participants contribute to or collect data for scientific research projects. Citizen science benefits both the researchers and the people engaged in it. Researchers can collect data that they otherwise would not be able to while the participants learn about the subject they are engaged with. For example, when using iNaturalist, an app that anyone can download on their phone, people collect data and learn about plant or animal species (Horn et al., 2018). Increasingly, citizen science platforms are going mobile with emerging technologies and shifting paradigms (Newman et al., 2012). To effectively engage in citizen science projects, the participants often need to develop skills for detecting, identifying, and annotating phenomena or entities from visual inputs. At present, it is expected that the participants already have these skill sets, or they can quickly develop these by following a set of instructions or tutorials (Rosser and Wiggins, 2018). However, for novice participants, it is often not as easy to understand some new phenomena or entities in real life just by following a set of instructions or tutorials. In this article, we use the conventional term ”researchers” to describe the group that runs the research projects and need to collect data, and ”participants” to describe the group that collects the data and contribute to the project using a citizen science platform or app (Eitzel et al., 2017).

To illustrate the challenges faced by potential participants in collecting research quality data, we describe the rip current detection problem (Philip and Pang, 2016). Rip currents are safety hazards that can claim human lives. To answer questions like: ”Are there rip currents at this beach?” the researcher needs to gather data that an army of participants can conveniently collect. However, spotting rip currents can be challenging for novice participants unless they are familiar with this subject matter (Brannstrom et al., 2015). Recent works demonstrated that rip currents can be detected using ML approaches (de Silva et al., 2021; Maryan et al., 2019). Providing real-time ML-based guidance using bounding boxes around rip currents in the live camera feed of the mobile app enables the participants to learn to spot rip currents and collect data more effectively. This facilitates the effective engagement of people who may have less familiarity with rip currents. The ML-based guidance can work as an educational tool for the participants as well, alerting them of potential danger. While this example focuses on rip currents, it can be replaced with a wide variety of fields and natural phenomena on which researchers are interested in collecting data, such as biological sciences, aquaculture, geomorphology, drought, and flooding indicators, to name a few. In all these cases, mobile apps with real-time ML-based guidance systems enable the participants to recognize and collect the correct data.

Indeed, there are similar infrastructures that allow one to create people-powered apps like those provided by Zooniverse, SPOTTERON, Anecdata, etc. (Liu et al., 2021). However, these general-purpose citizen science app builders are limited to providing standardized purpose-specific tools and features, such as task assignment and data uploading for crowdsourced research projects. Also, the data collection processes entirely rely on human skills as there is no integrated ML support in the client apps. Some apps like iNaturalist have server-side ML capabilities (Horn et al., 2018). However, to use server-side ML in real-time, continuous high-speed connectivity is required, which can be expensive, and internet connectivity is not available in many remote places where some projects might need to collect data. While considering building ML capabilities on top of the architectures of existing open-source systems (iNaturalist, Zooniverse, etc.), our analysis showed that they are not designed to integrate with client-side ML. It is possible to develop a new citizen science app from scratch with ML support for each project. However, developing and deploying each of these individual apps would take months, if not years. For example, an app with client-side ML capabilities to classify plant and animal species is Seek by iNaturalist (Horn et al., 2018). While iNaturalist already has a well-developed app with server-side ML, they had to create Seek from scratch to add client app-side ML support. While the topics of different citizen science projects may seem far afield from each other, common needs for collecting visual data tie these domain problems together. So, ML-powered apps created using a general-purpose citizen science platform can help participants recognize the correct visual data and increase the efficiency of the data collection process in a wide variety of research domains.

This article introduces an open-source software platform that allows a domain researcher to quickly create citizen science apps with integrated ML models to collect visual data, even if they don’t have a computer science background. Existing ML-powered citizen science apps often involve development stages that take months or years to deploy. Our proposed platform reduces many of those stages by providing a common feature set under a single framework shared by all apps. This will enable rapid prototyping and faster deployments allowing researchers without a large budget or projects that are more investigative than long-term in nature to engage in productive work.

We studied the most popular citizen science app creation platforms that allow one to create people-powered mobile apps. Zooniverse is a free citizen science web portal that allows creating projects for different domains (Barber, 2018). A sample project from Zooniverse is OceanEyes, where volunteers are sought to help count and label the fishes in the images that the researchers’ cameras have collected. It illustrates how having an ML model to identify the fish species can be highly beneficial. Anecdata is another free online citizen science platform that has similar features as Zooniverse (Disney et al., 2018). Another fully mobile app-based citizen science platform is SPOTTERON (Liu et al., 2021). All the citizen science apps using this framework have the same look and have an easy-to-customize GUI for various projects. Powered by SPOTTERON, another citizen science app is CoastSnap, which uses uploaded beach photos to understand how coastlines might change in the coming decades (Harley et al., 2018). App Movement is another authoring platform for community-created mobile apps, which provides automatic development and deployment of the app by customizing a common template (Garbett et al., 2016). However, the client-side ML component is not available on any of these platforms.

The main components of our proposed citizen science platform are the mobile app (middle), the ML models in the app (left), and the server-side components (right) shown in Fig. 1. The app contains the ML model and provides the primary interface for the participants. It provides standardized purpose-specific tools and features for crowdsourced research projects. Built-in tools include instructions, tutorials, data saving, uploading, etc. Since we aim to facilitate visual data collection, the app includes a camera tool with a live view that doubles as the visualizer for the ML model (e.g., bounding boxes around the detected objects). New projects initially start with a blank template with these built-in features, functionalities, and default look-and-feel that can be customized later.

For each project, the ML model needs to be trained with an initial dataset. We assume that the researcher has this. If a project has no data at all, the researcher’s team will need to collect some limited initial training data to create a rudimentary model that can be improved via continual learning as more data is collected (Schwarz et al., 2018). We also assume the researchers themselves and other domain experts will use the app as ”expert participants” who can collect higher quality data and provide labels that correct the misclassifications or false positive detections from the rudimentary model. Thus, there is an opportunity to engage the ”expert participants” more intimately by being part of the process to improve the ML model through confirmation or refutation. If a ”perfect” training dataset exists for a project, the researcher can directly use that for training the model and quickly start large-scale deployment for data collection.

The trained model is integrated with the app before building and deploying the app. When the participants use the app for data collection, the ML model runs with the app and guides with classifications or annotations (e.g., bounding boxes) to help them recognize the object for data collection. The models are fully compatible with mobile device architectures and run locally without internet connectivity and back-end server support. The back-end primarily works as the repositories for collecting the data that the participants upload. Other optional features included on the server-side are a companion website, user account, data management, data explorer, analysis and visualization, server-side ML apps with ML models requiring more computational power than mobile devices, etc. There is a primary server for managing all the apps for each citizen science project in our architecture. However, each project has its own data storage server (physical or cloud) for storing the collected datasets.

The architecture of our citizen science platform is a standard client-server system (Fig. 1). The app and the integrated ML model run on the client devices, e.g., smartphones. The phone camera provides real-time visual inputs for the ML model to process (Fig. 2). We use ML models based on TensorFlow Lite (Abadi et al., 2015). These models are small and optimized to run on limited computational resources on mobile devices. The same trained model runs on both Android and iOS versions of the app. At present, single-shot detector (SSD) models are supported by TensorFlow Lite. For our test cases, we used SSD MobileNetV2 (Sandler et al., 2018) and EfficientDet (Tan et al., 2020), and trained the models using transfer learning (Alsing, 2018).

Our platform simplifies the authoring process using a set of internal and external web-based tools. The fully guided app creation project starts on the website of our authoring platform. The website has the instructions and tools for the project creators to train a model using their custom dataset to bootstrap their project. Then, we provide another web-based tool to integrate the trained model and compile the app. Once the app is ready, it can be uploaded to the app distribution services for mobile devices to make it available for the participants to download and use.

Based on the guidance and feedback from the ML model running in the app, the participants can decide which data they want to capture. As the ML model integrated into the app runs locally on mobile devices, no server support or internet connectivity is required. The data is captured and initially stored in the local storage of the smartphone. The participants can later select the data they want to upload to the server. A larger ML model on the server-side can be used to analyze further and verify the collected data utilizing more powerful computational resources.

We did some initial testing of this architecture on two separate citizen science projects. The results from the two use cases are presented below.

This article presents an overview of an open-source platform for creating client-side ML-powered citizen science apps to improve data collection quality and efficiency. We demonstrate the use of the authoring platform with two real-world examples. As we continue working on the citizen science platform, we plan to optimize the overall process, including an enhanced user interface for mobile apps, support for a wider variety of ML models, and more server-side services.