Toward Among-Device AI
from On-Device AI with Stream Pipelines

R2 does not only imply using FlexBuffers (Google, 2021) for inter-pipeline transmissions. Data streams in a pipeline are often required to be schemaless (or dynamic schema). For example, a pipeline may pre-process a live video stream with an object detection neural network to crop the video, making a video stream filled with a detected human body. Then, the pre-processed tensor stream representing the cropped video may have varying dimensions per frame, which another neural network (e.g., pose estimation) may use as its input stream. Thus, even while schemaless data exchanges between pipelines might be useless, there is a need for the dynamic schema.

We update the tensor stream data type, “other/tensors” (MIME for GStreamer capability), so that users may specify the format of static, dynamic, and sparse. The conventional tensor stream with schema is static, and it is the default format. With dynamic format, the dimension and type may vary for each frame in a stream, which can serve schemaless data from remote pipelines and stream with dynamic schema. Unlike static format, which does not have format information in the buffers of a data frame, the dynamic format has a header in each data frame buffer that specifies dimension and type for each data frame. The other format, sparse, allows expressing tensors with the coordinate list format (COO) (SciPy, 2021).

Filters for tensor streams, tensor_* elements, are updated to handle static and dynamic formats. However, tensor_* elements do not directly handle sparse formats because its binary representation of tensor data is not compatible with the other two formats. Thus, we provide converting filters, tensor_sparse_enc and tensor_sparse_dec.

In addition to the Protocol Buffers and FlatBuffers support of the previous work, we add FlexBuffers of FlatBuffers for schemaless data transmissions as sub-plugins of tensor_converter (FlexBuffers to “other/tensors”) and tensor_decoder (“other/tensors” to FlexBuffers).

There are various raw network protocol stream filters as off-the-shelf shared libraries of GStreamer. According to GStreamer’s reference page at https://gstreamer.freedesktop.org, they include TCP, UDP, RTSP, HTTP, HTTPS, FTP, HLS, and many others with different serialization mechanisms. However, as explained in the previous section, they are not enough for among-device AI applications while they appear almost complete for multimedia applications and services.

NNStreamer-Edge is a lightweight and portable library implementing the proposed protocols to connect with NNStreamer pipelines without adopting NNStreamer. NNStreamer-Edge is an open source software package independent from NNStreamer and its basis, GStreamer: https://github.com/nnstreamer/nnstreamer-edge. It does not depend on NNStreamer or GStreamer so that devices that cannot afford GStreamer or heavy operating systems may easily use NNStreamer-Edge. NNStreamer-Edge has minimal library dependency, Paho MQTT C client library. This library also has minimal library dependency; if SSL is disabled, it only requires essential tools (e.g., GCC and libc).

NNStreamer-Edge of October 2021 supports tensor stream publish for remote sensors and cameras: “edge_sensor” module. This module behaves like an “mqttsink” NNStreamer element with “other/tensors” streams to connect with “mqttsrc” NNStreamer elements. We have designed other modules of NNStreamer-Edge, “edge_output” and “edge_query_ client”, but have not yet released them.

The primary objective is to extend connectivity for devices of arbitrary vendors; thus, the proposed mechanisms should be compatible with or included in alliances such as Matter (Connectivity Standards Alliance, 2021) and SmartThings (SmartThings, 2021). Because NNStreamer-Edge has chosen Apache 2.0 license and does not have extra dependencies, anyone may implement their proprietary software with NNStreamer-Edge. For example, third-party developers may implement a proprietary MediaPipe plugin with NNStreamer-Edge so that arbitrary MediaPipe pipelines may communicate with NNStreamer pipelines. Note that DeepStream pipelines are GStreamer pipelines; thus, they can trivially connect to NNStreamer pipelines, which are also GStreamer pipelines.

This section describes how the proposed implementation satisfies each requirement mentioned in the first section.

1. R1.

   As long as we implement AI services and shared input/output streams as NNStreamer pipelines, we can deploy them atomically as pipeline instances or pipeline descriptions. With tensor_query_client/server, we can offload inference tasks to another pipeline. With mqttsrc/sink, a pipeline may serve as a data stream publisher or subscriber.

2. R2.

   If a stream is typed as “other/flexbuf” using “Flexbuf” sub-plugins, it is schemaless. A NNStreamer-native stream can be schemaless if it is typed as “other/tensor,format=flexible”. The former is for compatibility with third-party software, and the latter is for in-pipeline or inter-pipeline streams.

3. R3.

   The adoption of MQTT for connections has satisfied this requirement. Multiple subscribers or clients can connect with a server with topic names. Multiple publishers or servers may be available for a given client or subscriber to choose with topic filters and wildcards. The timestamp synchronization mechanism described in Section 4.2.3 supports synchronization. Sparse tensors and gst-gz (Esse, 2021) support compressed transmissions.

4. R4.

   MQTT client libraries and MQTT brokers handle this.

5. R5.

   NNStreamer is LGPL 2.1 with all source codes included as default plugins to satisfy this requirement. For systems without NNStreamer and GStreamer, but with their proprietary middleware, we release NNStreamer-Edge with Apache 2.0 so that users may distribute their software without the condition of LGPL 2.1. Anyone may participate in designing and developing both packages in open space, Github.com.

6. R6.

   NNStreamer is cross-platform and deployed for different operating systems officially: Tizen, Android, Ubuntu, Yocto, and macOS. In addition, users have reported using it in Debian and OpenSUSE as well. It would not be too difficult to port it for Windows or iOS, but we have not tried it yet. The inter-pipeline connectivity library, NNStreamer-Edge, is designed to make the among-device AI capability compatible with different operating systems and different pipeline frameworks.

7. R7.

   We satisfy this by allowing developers to implement AI services with conventional NNStreamer plugins and adding inter-pipeline transmission plugins. For example, in Tizen, adding among-device AI capability (Tizen 6.5 M2) does not break the backward compatibility of the machine learning API set.

Listing 1 shows a script code representing GStreamer pipelines in Figure 2. GStreamer API or its wrapper, NNStreamer API (Tizen ML API), can execute such a script. We can also execute the script directly on a shell with “gst-launch” for prototyping and testing. We can apply such pipelines to home IoT systems consisting of a device with comfortable user interfaces without high computing power (e.g., an inexpensive TV) and a device with less comfortable user interfaces with high computing power (e.g., a home IoT hub or a mobile phone). In other words, if a user has an inexpensive but large display (Device A) and a high-end mobile phone (Device B). The user may run home fitness and training applications by offloading a pose-estimation neural network to Device B while using the camera and screen of Device A.

Please note the simplicity of server (Device B) code; declaring the service name (${SVCNAME}) is all developers need to do. If users want more operations in servers–e.g., pre-processing or data collecting–, they may add corresponding filters to the pipeline. The client-side pipeline is straightforward as well. The only changes from its on-device AI equivalent pipeline are replacing tensor_filter with tensor_query_client and declaring the service name. Then, we may have multiple clients and multiple servers for the given service name for resource sharing and robustness.

The string variables in Listing 1 depend on neural network models and their input and output formats. For example, if it is a Mobilenet V2 object detection model (Sandler et al., 2018) from TensorFlow Hub with COCO 2017 dataset (TensorFlow, 2021), the variables are:

Configurations and behaviors of queues and merging points (compositor in this example) are crucial for the efficiency of parallelism. With the leaky=2 option, a queue drops older buffers if it becomes full. Users may alter options, including the size of the queue and leaky modes, for further optimization.

Listing 2 shows a GStreamer pipeline description of the application example shown in Figure 3, where FlexBuffers (Google, 2021) serializes published streams. When the v4l2src creates a video frame buffer, fetching video streams from a USB camera attached to a Raspberry Pi 4 board in our experiments, the v4l2src provides a timestamp value with the option of do-timestamp=true to enable the mechanism in Figure 4. Note that this code does not require the camera to be a USB camera or the system to be a Raspberry Pi 4. The same code works for an embedded camera of a mobile phone or a typical Linux desktop PC without modification. If multiple cameras are attached to the system, users may specify it with additional options; otherwise, it uses /dev/video0.

The pipeline topology of the target application in Figure 3 has become more complex than the application in Figure 2. It would require a considerable amount of time and effort to implement such systems without pipeline frameworks as we have experimented in (Ham et al., 2021); i.e., it would probably require well over thousands of lines of codes. With the off-the-shelf GStreamer/NNStreamer plugins and effortlessly applied pipe-and-filter architecture, users can write such an among-device AI system within 100 lines of codes, which is easy to extend and port. We can execute the same pipeline descriptions in Ubuntu PC, Yocto devices, Tizen devices (TVs, home appliances, robots, and wearable devices), or Android mobile phones. With the NNStreamer-Edge library, developers can interconnect pipelines with more varying devices, including MediaPipe/DeepStream pipelines running in clouds or workstations and lightweight devices (microcontrollers) running RTOS.

Note that FlexBuffers, FlatBuffers, or Protocol Buffers are not required to connect NNStreamer pipelines and NNStreamer-Edge instances only. For other independent non-NNStreamer processes subscribing to the published streams, we can apply FlatBuffers, FlexBuffers, or Protocol Buffers, popular data serialization mechanisms.

Another example, Figure 5, shows an augmented worker application, which is both multi-device and multi-modal. Such a system may assist workers in manufacturing plants by detecting and notifying events requiring attention. For example, if a worker assembles parts incorrectly, the system sends an alarm to the worker.

In the left-hand side pipeline of the mobile device, the “DETECT” model detects if an action of assembling parts is starting and let the wearable device know. Then, the wearable device will start streaming related data from the microphone and IMU sensors back to the mobile device. Then, the right-hand side pipeline of the mobile device decides whether the assembling activity is correct or incorrect based on the data from the wearable device and reports to the application logic. To further optimize power consumption in the wearable device, we may turn on and off the sensors based on the “activation” signal from the mobile device. As we demonstrate with previous examples, the pipeline description code of this example incurs short lines of codes as well; i.e., usually a single line per block with exceptions of tensor_if, where we need to describe the condition with a script or a C function.

We evaluate the proposed among-device AI transport mechanisms: MQTT pub/sub and MQTT-hybrid query along with their lighter and faster counterparts: ZeroMQ pub/sub and TCP query. We have experimented with Raspberry Pi 4 boards connected via Ethernet and NNStreamer 2.1.0-unstable. Figure 6 shows the evaluated among-device AI pipelines. Case A evaluates the stream pub/sub of MQTT and ZeroMQ. Case B evaluates the query client/server streams of MQTT-hybrid and TCP direct connections. We measure the throughput, CPU usage, and peak memory consumption to evaluate both performance and overhead. We experiment with three different bandwidths of input streams from Device A and Device C: high, mid, and low bandwidths. Each bandwidth corresponds to a Full-HD video stream, a VGA (640x480) video stream, and a QQVGA (160x120) video stream with a 60 Hz framerate.

Figure 7 shows the evaluation results of MQTT pub/sub and MQTT-hybrid query, normalized by their counterparts, ZeroMQ and TCP, respectively. Bars in the figure are normalized standard deviations, and values average five experimental runs of 30 seconds. Case M and H have failed to transmit 60 Hz, implying that the network bottlenecks the throughput. MQTT suffers from lower throughput and higher client memory overhead with high bandwidth streams (M and H). MQTT-hybrid successfully eliminates the performance overheads of MQTT while keeping the rich features of MQTT. For example, with the MQTT-hybrid query, if multiple servers are compatible with the client’s topic, the client pipeline is automatically switched to another server when a connected server becomes unavailable. Moreover, server pipelines may declare additional specifications for clients to choose, e.g., “server workload status” and “neural network model and version”. Note that the query plugins have both TCP direct and MQTT-hybrid implementations switchable at run-time. For higher performance, we plan to add MQTT-hybrid for pub/sub plugins, as well.

After wide deployment of the previous work (Ham et al., 2021) across many prototypes and products, including mobile phones, wearable devices, TVs, and home appliances, we have worked on prototypes of various internal clients based on the proposed among-device AI methods. As a result, we have observed the following lessons and future work.

Many users appear to feel barriers against adopting pipe-and-filter architecture. It is easy to show that pipelines with NNStreamer work appropriately and significantly better than the conventional implementation for both developmental costs and run-time performance. However, we could have observed unexpectedly steep learning curves to adopt pipeline concepts and to describe pipeline topology. For the former issue, we are preparing to write more diverse pipeline examples for users. For the latter issue, we are implementing a WYSIWYG pipeline editor with a converter translating between GStreamer scripts and MediaPipe scripts (pbtxt) to re-use MediaPipe’s pipeline editor.

Analyzing and profiling pipeline performance becomes more complicated with among-device AI pipelines. We have an AI pipeline profiling tool, nnshark, which forks GstShark (Gruner, 2017), created by a group of undergraduate students as an open source software project. We have ported nnshark for Tizen and Android devices and deployed it to users. However, with among-device AI capability, users are not satisfied with nnshark, and request profiling capability for the whole system consisting of multiple pipelines simultaneously.

We have been implementing initial prototypes or practicing pair programming for internal clients to address learning curves and developer relations. For external users of open source communities, we try to catch up with technical questions in various channels (Slack, Github issues, mailing lists, and direct contacts) in addition to documents and sample applications. However, with frequent requests from internal clients and our roadmap of new features, writing documents and sample applications have always had lower priority and been behind schedule. Besides, with a few core contributors focusing on code contributions, it is challenging to motivate writing documents. It is also difficult to justify having a dedicated document writer while we directly support internal clients so that they do not even bother to read the documents.

NNStreamer has external users that have contacted the authors: NXP Semiconductors, Collabora, ODKMedia, and fainders.ai. Feedback and contributions from them are constructive for identifying issues and requirements. Motivating external users to communicate with maintainers is essential; they are often too shy to do so. We recommend keeping such communication in public channels to motivate other users to communicate with maintainers and the community.

Users often write pipelines incorrectly or inappropriately, and it is usually too late when we find it out. For example, an NNStreamer application (Lee et al., 2020) drawing bounding boxes of detected objects with live video streams has an inappropriate pipeline design. Creating video streams from neural network output is supposed to be handled by tensor_decoder. Its authors (Lee et al., 2020) are aware of it; however, for neural network output formats not supported by default sub-plugins of tensor_decoder, they have abandoned tensor_decoder and made their application thread directly decode outputs and push decoded data into the pipeline. It is supposed to write a tensor_decoder sub-plugin for such a case, which incurs lower overhead, higher throughput, and less implementation effort. Fortunately, although inappropriately implemented, their NNStreamer implementation is shown to beat Open CV implementation with higher throughput. Promoting communication with the NNStreamer community may address this issue; however, we have observed similar cases with internal users: the users have sometimes released inappropriate pipelines before addressing them. Such releases are especially troubling considering that we are still at an early stage of deploying the pipeline paradigm to AI developers in the affiliation. Besides, the standard practice of software verification and testing is not ready for such issues. We may need another verification and testing process for newly introduced frameworks with a new concept, allowing framework developers to audit. Such processes will provide more usage cases for framework developers and more chances to improve their frameworks.

NNStreamer has started evolving for among-device AI systems. Besides the apparent performance optimization, we have suggestions for among-device AI systems and pipeline frameworks as future work.

With frameworks including NNStreamer (Ham et al., 2021), DeepStream (Nvidia, 2021), and MediaPipe (Lugaresi et al., 2019), developers have started writing AI systems as pipelines. These frameworks provide various documents and sample pipelines to mitigate difficulties suffered by developers writing AI pipelines. However, as our users have complained, they are not enough.

We would like to suggest another method directly intervening pipeline development process. First, we need to provide common parts of pipelines (sub-pipelines) as libraries; that developers can invoke or insert sub-pipelines in their pipelines. There are various common parts in different AI applications: e.g., pre-processing video streams for object detection or audio streams for RNN-T (Rao et al., 2017). This feature may often prevent inappropriately designing pipelines as well.

Then, we need a pipeline run-time repository where processes may register pre-defined pipelines, and other processes may invoke such pipelines. This feature enables operating systems or middleware to register pipelines for applications. It enables applications without particular AI features to invoke such pipelines without actually writing pipelines. Moreover, suppose a vendor wants to separate the application development and AI development divisions; this approach will cleanly separate code repositories for corresponding divisions as a client has wanted.

We have further future directions that require more time and effort; the above can be implemented and deployed within a year. We envision an IoT ecosystem where devices of any vendors may join, which is related to R5 and R6. Matter (Connectivity Standards Alliance, 2021) along with SmartThings (SmartThings, 2021) is a promising candidate for such an IoT ecosystem; however, it lacks a data transmission protocol for inter-device AI services. In the future, we expect to propose such a protocol with among-device AI capabilities mentioned in this paper with extensions of interoperability with microcontrollers and other pipeline frameworks. With such protocols included in the IoT ecosystem, connected devices will be able to consistently provide AI services regardless of interface devices’ computation power. This inclusiveness and consistency will promote the proliferation of various on-device and among-device AI services without the need for exposing privacy and private data to external computing nodes.