On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Study

The paper presents a real-world case study of a target classification application, based on seismic and acoustic sensing, that demonstrates how a self-supervised neural network model pre-trained with unlabeled sensor data (using pre-training techniques common to foundation models [1]) can significantly improve run-time inference robustness and adaptation. Modalities, such as acoustic or seismic sensing, are particularly sensitive to environmental factors. Even in the same application domain, such as target tracking, a target (e.g., some vehicle on a road) may generate different sensory signatures depending on a variety of factors, such as the type of terrain (paved road, gravel, sand, …), background noise (rain, wind, construction, traffic, …), and other natural and/or human disturbances. Training an inference task (e.g., a target classifier) to handle all such contingencies is a daunting undertaking. Inspired by pre-training solutions used for foundation models, can one pre-train a general target-independent and environment-independent model once, based on large amounts of unlabeled data (henceforth called a foundation model), then fine-tune it in a very light-weight fashion to each deployment environment and set of targets of interest?

Early supervised solutions for intelligent IoT applications are label-hungry due to the large sizes of modern deep neural networks (DNNs) that call for commensurately large volumes of (labeled) input training data. In the absence of sufficient amounts of labeled data, supervised neural-network training techniques suffer from overfitting, thereby dramatically reducing the robustness of run-time inference [2]. In contrast, by obviating the need for labeled data in pre-training (and requiring only small amounts of labeled data for fine-tuning), foundation models developed for intelligent IoT applications can improve inference robustness and adaptation to domain shifts and environmental noise.

Unlike supervised training techniques that directly teach a neural network how to perform a particular inference task, a foundation model is the output of (pre-)training that aims to teach the neural network a better internal representation of domain-specific data. By empirically learning statistical properties and patterns found in large domain-specific datasets, such an internal representation encodes (empirical approximations of) higher-level semantics or “knowledge” of the domain. Clearly, the degree to which such outcomes can be elicited depends on the amount of data used. Three important features thus characterize the pre-training of foundation models. First, it is self-supervised; no labeled data are needed. Second, it is task-agnostic; it does not know the downstream inference task(s) and, as such, can in principle support several different tasks, deployments, or environments. Finally, it generally uses a large amount of (unlabeled) data. For the sake of a proof of concept, we sacrifice the last property a bit in this study. The feasibility of pre-training in the absence of labeled data and without knowing the exact downstream task(s) makes the approach attractive to IoT applications, especially from a robustness perspective. Interestingly, despite the use of a smaller (and thus more manageable) amount of pre-training data in this paper, the robustness advantages of the resulting model are still possible to illustrate. We show that the pre-trained model can be fine-tuned with only a minimal amount of labeled data for a specific downstream deployment, allowing more robust classification than baseline (supervised) approaches.

Another challenge for IoT sensing is the computational limitations in IoT devices. Rapid advances in computational resources have led to increasingly large DNNs [3]. However, many IoT devices remain limited by their resource constraints [4]. These devices, from simple sensors to complex wearables, often lack the necessary processing power, memory, and energy efficiency to support the training and operation of large-scale DNNs in real-time. This discrepancy poses significant challenges for deploying and training advanced DNNs in IoT applications [5], introducing bottlenecks to the model performance on IoT devices. We show that the pre-trained model we use is capable of execution on a Raspberry-Pi class of devices in real-time with a higher fine-tuning convergence rate while offering more robust performance than its supervised counterparts.

The rest of the paper is organized as follows. We cover a brief background on foundation model pre-training and the specific model used in this paper in Section II. We describe our case study and experimental set-up in Section III. Section IV presents the evaluation results, followed by discussion in Section V. Section VI covers related work. Section VII concludes the paper.

While many techniques were proposed recently for self-supervised pre-training of foundation models, two are particularly widespread: learning to reconstruct masked [6, 7] (or distorted [8]) inputs and contrastive learning [9, 10, 11, 12]. They differ in the way they train the model useful concepts from the domain, without the need for labeled data. Specifically, masking/distortion removes/distorts parts of the input, and then rewards the model for the correct reconstruction of these parts. Clearly, a model that learns correct reconstruction must have encapsulated some knowledge about the target domain. Contrastive learning teaches the model what “similarity” means in the target domain (by contrasting similar and dissimilar sample pairs), such that similar inputs are grouped closer together in a latent space. To do so without labels, it often relies on semantics-invariant input transformations that convert individual input samples to “similar” ones (without necessarily knowing what the sample labels or semantics are). An example of such transformations in vision is image resizing. An example in time-series data is adding simulated noise. The result of rewarding the model for putting similar samples closer together in the latent space is a well-organized learned latent representation, where proximity implies semantic similarity.

In this paper, for pre-training, we use a contrastive learning framework, called FOCAL [1], recently proposed for (pre-training in) intelligent multimodal sensing applications. FOCAL pre-trains an encoder to extract a structured latent representation of the input multimodal sensing data. This latent representation separates shared and private subspaces. The shared subspace contains common information shared across the different sensing modalities. The private subspaces hold additional modality-exclusive information. An orthogonality constraint is applied among the private subspaces, as well as between each private subspace and the shared subspace to enforce information independence among these subspaces. A pre-trained encoder is fine-tuned by appending a single linear layer whose weights are adapted to the downstream use scenario (using a small amount of task-specific labeled data).

We utilize FOCAL to train two popular DNN encoders (DeepSense [13] and SWIN-Transformer [14]) on a multi-modal Moving Object Detection [1] (MOD) dataset that consists of acoustic and seismic signals. Then, we perform a two-day experiment in a real-world neighborhood as a case study to examine the performance of FOCAL against supervised counterparts. The experimental setting and results are described below.

Our experiment was conducted at an outdoor research facility located on (repurposed) state park grounds. Sensors were deployed and vehicles were driven past the sensors over a period spanning two days. On the first day, the environment surrounding the experiment was controlled and disturbance-free. The second day featured significant interference (as described in the next section).

FOCAL was pre-trained on a previously published dataset [1] collected from acoustic and seismic sensors, deployed in different urban and rural environments that varied in terrain (paved, gravel, dirt, rooftop, etc) and environmental conditions (quiet, windy, etc), recording the passage of a variety of target types, mostly focusing on civilian automobiles, bikes, and humans. The pre-training data did not include any from the deployment reported in this paper. To experiment with the robustness of the pre-trained model, we fine-tune it on part of the data collected in the new deployment and test the fine-tuned model’s performance under the same or different deployment conditions. A comparison is carried out with supervised approaches.

We show that FOCAL is more robust to domain changes than other methods. Additionally, we demonstrate FOCAL’s superiority in terms of label efficiency by comparing performance under different amounts of labels used for fine-tuning. Finally, we present our findings on the computational efficiency of FOCAL and explore the potential applications for real-time IoT systems.

Below, we examine FOCAL performance after fine-tuning with some target domain data then compare the computational efficiency of the supervised and the Foundation models.

The experimental study reported in this paper suggests that the task-agnostic nature of pre-training of self-supervised models endows them with greater robustness, making them ideally suited for IoT application deployment across various environments with only limited fine-tuning needed to achieve high-quality inference. Unlike traditional supervised models, these pre-trained models exploit large amounts of unlabeled data, leading to enhanced resilience against domain shifts. Although this paper only leverages a small-scale unlabeled dataset for pre-training, the pre-trained models already exhibit characteristics of foundation models with great robustness in different domains. This is particularly useful in IoT sensing scenarios where different sensor deployments (even within the same application) may be subjected to vastly different conditions. For example, target tracking using acoustic/vibration sensing will see significant distributional shifts across urban areas, rural roads, freeways, gravel parking lots, etc, as well as across different weather conditions (wind, snow, rain), and different target types. Pre-training the foundation model with a larger scale dataset and larger backbone encoders could also potentially improve its downstream robustness, and we leave that to future work.

The high label efficiency of pre-trained models further facilitates their rapid deployment to a wide array of downstream tasks, where label scarcity is a critical challenge. These models exhibit exceptional adaptability to varying physical environments, which makes them suitable to meet the demands of real-time CPS. Merely training a single linear layer in FOCAL, for fine-tuning, can easily reach optimal performance within a few epochs. This efficiency not only enhances the practicality of FMs in dynamic settings but also opens opportunities for on-device training, making it feasible to train FMs on resource-constrained IoT devices.

Deep Learning has catalyzed significant advances in inference from IoT sensing data [20], with DNNs becoming integral to a wide range of IoT applications [21, 22]. However, domain-specific challenges still lead to many limitations in building robust DNNs for IoT sensing. Deployed DNNs must handle unpredictable interference in the field that greatly alters the statistical distribution of collected sensor data. The altered distribution, or domain shift, can significantly degrade DNN performance, leading to inaccurate results and potentially severe consequences.

More recently, Foundation Models (FMs) [23] have gained increasing popularity, most notably in language [6, 24] and vision [7, 25]. The techniques were then generalized to other areas where domain-specific FMs emerged, such as security [26, 27], networking [28, 29], and meteorology [30].

Contrastive Learning (CL) [1, 31, 32, 33, 34, 35] has been a popular form of SSL to extract a robust embedding space during pre-training. The main idea is to pull similar samples closer while pushing other samples further apart in the embedding space. Unimodal CL frameworks like [36, 31] apply random augmentations to learn transformation invariant information. [37, 33, 32] are multi-modal CL frameworks that enforce cross-modal consistency. CL for time series has also been extensively studied in [34, 35].

Improving resilience against domain shifts has been widely studied in recent years. [38, 39] investigate improving the efficiency of unsupervised domain adaptation for IoT applications. However, these works primarily consider classifiers trained in a supervised manner. Others have also worked on Federated Learning-based domain generalization [40, 41]. Numerous works analyze SSL for domain generalization [42, 43], but less has been explored for IoT applications.

In this paper, we examined an FM-based approach, specifically FOCAL, against conventional supervised models in the context of IoT sensing. Through our real-world case study, we have demonstrated how Foundation Models require minimal domain-specific tuning while allowing robust real-time inference. Our results highlight promising opportunities for Foundation Models in the IoT landscape. Our future work will focus on developing more scalable Foundation Models for generalized IoT systems.

Research reported in this paper was sponsored in part by DEVCOM ARL under Cooperative Agreement W911NF-17-2-0196 (ARL IoBT CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S.Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.