Diverse Intra- and Inter-Domain Activity Style Fusion for Cross-Person Generalization in Activity Recognition

In cross-person activity recognition (Qin et al., 2023), a domain is characterized by a joint probability distribution $P_{X,Y}$ across the product space of time-series instances $\mathcal{X}$, and the corresponding label space $\mathcal{Y}$. Each instance $\mathbf{X}_{i}\in\mathbb{R}^{K\times L}$ represents the values of each time series obtained from sensors, where $K$ is the dimensionality of features, and $L$ is the temporal length of the series. Moreover, each instance $\mathbf{X}_{i}$ corresponds to an activity class label $y_{i}\in\{1,2,\dots,C\}$, indicating the specific activity category performed by the subjects, with $C$ denoting the total number of activity categories. The domain generalization challenge lies in the nonintersection and domain differences between the training and testing sets. Typically, the training set $D^{s}=\{(\mathbf{X}_{i},y_{i})\}_{i=1}^{n^{s}}$ is collected from the labeled source-domain subjects, where $n^{s}$ represents the number of training instances. Importantly, $n^{s}$ is often small in cross-person activity recognition scenarios, presenting the small-scale challenge. On the other hand, the test set $D^{t}=\{(\mathbf{X}_{i},y_{i})\}_{i=1}^{n^{t}}$ consists of $n^{t}$ instances obtained from the unseen target-domain subjects and satisfies the condition $D^{s}\cap D^{t}=\emptyset$. In addition, source and target domains have different joint probability distributions while sharing the identical feature space and class label space, i.e., $P^{s}(\mathbf{X}_{i},y_{i})\neq P^{t}(\mathbf{X}_{i},y_{i})$, and $\mathcal{X}^{s}=\mathcal{X}^{t}$, $\mathcal{Y}^{s}=\mathcal{Y}^{t}$. The primary objective is to leverage the available data in $D^{s}$ to train a TSC model $f:\mathcal{X}\rightarrow\mathcal{Y}$ capable of effectively generalizing to an inaccessible, unseen test domain $D^{t}$, without any prior exposure to target domain data or domain labels during training. This task is inherently more challenging than conventional transfer learning settings (Qin et al., 2023; Qian et al., 2021) due to the disparate distributions across source and target domains, compounded by the small-scale settings of the training data.

Diffusion model (Sohl-Dickstein et al., 2015) involves training a model distribution $p_{\theta}(x)$ to closely approximate the target ground-truth data distribution $q(x)$. It assumes distribution $p_{\theta}(x)$ as a Markov chain of Gaussian transitions: $p_{\theta}(x_{0})=\int p_{\theta}(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t})dx_{1:K}$, where $x_{1},\dots,x_{T}$ denote the latent variables with the same dimensionality as original (noiseless) data $x_{0}$. $p_{\theta}(x_{T})\sim\mathcal{N}(0,\textbf{I})$ is the Gaussian prior. $p_{\theta}(x_{t-1}|x_{t})$ is the trainable reverse process given by

Diffusion predefines a forward process that progressively adds Gaussian noise to $x_{0}$ in $T$ steps, defined as

where $\beta_{t}\in(0,1)$ is variance schedule for noise control.

Training procedure.   The diffusion model’s parameters $\theta$ are optimized by maximizing the evidence lower bound of the log-likelihood of the data, i.e., $\log p_{\theta}(x_{0})$, which can be further simplified as a surrogate loss (Ho et al., 2020):

where $\mathcal{U}$ is the uniform distribution and the noise predictor ${\epsilon}_{\theta}(x_{t},t)$, parameterized with a deep neural network, aims to estimate the noise $\epsilon$ at time $t$ given $x_{t}$. As $\mu_{\theta}(x_{t},t)$ is determined by ${\epsilon}_{\theta}(x_{t},t)$, the target $p_{\theta}(x_{t-1}|x_{t})$ can be consequently derived.

Sampling procedure.   Given a well-trained $p_{\theta}$, the data generation procedure begins with a Gaussian noise $x_{T}\sim\mathcal{N}(0,\textbf{I})$ and proceeds by iteratively denoising $x_{t}$ for $t=T,\dots,1$ through $p_{\theta}(x_{t-1}|x_{t})$, culminating in the generation of the new data $x_{0}$.

Conditional diffusion models are typically guided by label or text prompts that provide task-specific knowledge, such as “create a [cartoonish] [cat] image” (Min et al., 2023; Zhang et al., 2022b; Wang et al., 2022b). However, the generation of instance-level time-series data introduces distinct challenges. It is difficult to capture the complex patterns solely through label or text prompts due to the inherently high-dimensional and non-stationary nature (Huang et al., 2020b). To address this issue, we propose the development of a style conditioner using a contrastive learning approach (Eldele et al., 2021). This approach has demonstrated robustness in extracting representations from unlabeled time-series data. The transformed data can retain the distinctive characteristics of the original data while preserving the semantic information of the classes. Thus, it is well-suited for extracting robust instance-level representation, termed as “style”, which can serve as conditions to guide diffusion models.

Delving into specifics, the contrastive learning pipeline consists of a feature encoder and Transformer on the available training data. The objective is to maximize the similarity between different contexts of the same sample and minimize the similarity between contexts of different samples. Once the module is trained, we utilize it as style conditioner denoted as $f_{\text{style}}$. When extracting the style from the original data $\mathbf{X}_{i}$, the style conditioner produces a style vector $S_{i}=f_{\text{style}}(\mathbf{X}_{i})\in\mathbb{R}^{H}$, where $H$ denotes the length of the vector. Consequently, each activity style condition can be interpreted as “a [$y_{i}$] activity performed in [${S}_{i}$] style”, where $y_{i}$ denotes the class of the original data. This approach takes an important step towards the first criteria of domain padding due to the preservation of class semantics. The aggregation of all context vectors from $n^{s}$ training instances constitutes a set $\mathcal{S}=\{{S_{i}}\}_{i=1}^{n^{s}}$, which can be further divided into $C$ class-specific subsets corresponding to $C$ classes. Each subset contains style vectors pertaining to a specific class, expressed as $\mathcal{S}=\{\mathcal{S}^{1}\cup\mathcal{S}^{2}\cup\cdots\cup\mathcal{S}^{C}\}$. In Appendix A, we provide the details of the contrastive learning approach (Eldele et al., 2021).

To control the generation of time-series samples, we can leverage the style in $\mathcal{S}$ to guide the conditional sampling process $p_{\theta}(\tilde{x}_{t-1}|\tilde{x}_{t},s)$ presented in Eq. (4). To this end, we adopt the classifier-free guidance (Ho and Salimans, 2022), which has proven to be effective in generating data with specific characteristics. In this framework, the training process is modified to learn a conditional $\epsilon_{\theta}(\tilde{x}_{t},t,s)$ and an unconditional $\epsilon_{\theta}(\tilde{x}_{t},t,\emptyset)$, where $\emptyset$ symbolizes the absence of the condition $s$. The loss function is formulated as follows:

where condition $s$ is one style feature in $\mathcal{S}$ derived from the pre-trained conditioner, and it is randomly dropped during the training.

During the sampling phase, a sequence of samples $\tilde{x}_{T},\dots,\tilde{x}_{0}$ is generated starting from $\tilde{x}_{T}\sim\mathcal{N}(0,\textbf{I})$. For each timestep $t$, the model refines the process of denoising $\tilde{x}_{t-1}$ based on $\tilde{x}_{t}$ through the following operation:

where $\omega$ is a scalar hyperparameter that controls alignment between the guidance signal and the sample (Ho and Salimans, 2022). Through the iterative application of Eq. (6), the diffusion model is capable of sampling new time-series samples that conform to specific styles $s\in\mathcal{S}$. It is worth noting that the styles $S_{1},\dots,S_{n^{s}}$ are robust to the class labels, the generation process guarantees the first criterion of domain padding: each generated sample belongs to a known class under the guidance of a single condition.

So far our approach has not yet achieved the second criterion of domain padding, as samples conditioned on a singular style $s\in\mathcal{S}$ could demonstrate a limited range of variation within the intra-domain space. Therefore, we then propose a style-fused sampling strategy to further enhance the diversity. This strategy guides the diffusion to generate new data that satisfy any number and combination of styles (conditions), rather than just one. By doing so, the generated data fuse diverse inter and intra-domain styles, effectively meeting the second criterion of domain padding.

Random style combination.   Random style combination entails the ensemble of one or multiple style features under a unified class to establish a new diffusion sampling condition. Importantly, the ensemble styles must belong to the same class to preserve class consistency. For each class label $c$, we randomly select any number of style features from the class-specific set $\mathcal{S}^{c}$ and combine them in all possible ways. This will end up with $2^{k}-1$ different style combinations (excluding the empty set), where $k=|\mathcal{S}^{c}|$ is the number of styles in $\mathcal{S}^{c}$. Mathematically, the collection of all possible style combinations for class $c$ can be expressed by the power set $\mathcal{P}(\mathcal{S}^{c})$ of $\mathcal{S}^{c}$:

For instance in Fig. 2 (c), three styles in $\mathcal{S}^{c}=\{S_{1},S_{2},S_{3}\}$ results in 7 different combinations¹¹1$\mathcal{P}(\mathcal{S}^{c})=\big{\{}\{S_{1}\},\{S_{2}\},\{S_{3}\},\{S_{1},S_{2}\},\{S_{1},S_{3}\},\{S_{2},S_{3}\},\{S_{1},S_{2},S_{3}\}\big{\}}$. This operation is replicated across all classes $1,\ldots,C$, integrating them into a comprehensive style combination set $\mathcal{D}=\{\mathcal{P}(\mathcal{S}^{1})\cup\dots\cup\mathcal{P}(\mathcal{S}^{C})\}$. The randomness in selecting style combinations can significantly foster diversity within and between domains, and maximize the exploitation of existing styles to generate highly diverse condition space $\mathcal{X}^{\text{cond}}$.

Style-fused sampling.   Subsequently, our efforts are directed towards empowering the diffusion model to fuse multiple styles during the data generation conditioned on a specific style combination $\mathcal{D}_{j}\in\mathcal{D}$. Assuming the diffusion has learned the data distributions $\{\epsilon_{\theta}(\tilde{x}_{t},t,s)\}_{i=1}^{n_{s}}$ through Eq. (5), sampling from the composed data distribution $q(\tilde{x}_{0}|\mathcal{D}_{j})$ for any given style combination $\mathcal{D}_{j}\in\mathcal{D}$ is achieved using the below perturbed noise:

The derivation of Eq. (8) is provided in Appendix B. This indicates that while the diffusion training process primarily focuses on an individual style, we can flexibly combine these styles during sampling. For instance, consider the combination of $\mathcal{D}_{j}=\{S_{1},S_{3}\}$ in Fig. 2 (c) and (d). Each element represents a style associated with the [walking] activity. Eq. (8) can generate new samples with the [walking] label possesses unique characteristics that fuse these two styles. This is critical for inter and intra-domain diversity in domain padding: diffusion can flexibly incorporate class-specific instance-level styles from different or the same domains to generate new samples with novel domain distribution. Moreover, given the existence of sub-domains within each domain, our diffusion model is capable of synthesizing novel domains, even from sampling instances within the same domain (we verify this later in the experiments).

Finally, we elaborate on the comprehensive workflow of our approach, which we refer to as Diversified Intra- and Inter-domain distributions via activity Style-fused Diffusion modeling (DI2SDiff).

Architectural design.   The diffsuion model $\epsilon_{\theta}:\tilde{\mathcal{X}^{s}}\times\mathbb{N}\times\mathcal{X}^{\text{cond}}\rightarrow\tilde{\mathcal{X}^{s}}$ is built upon a UNet architecture (Ho and Salimans, 2022) with repeated convolutional residual blocks. To accommodate the characteristics of time series input, we adapt 2D convolution to 1D temporal convolution. The model incorporates a timestep embedding module and a condition embedding module, each of which is a multi-layer perceptron (MLP). The condition embedding module is used to encode each activity style $s\in\mathcal{S}$, and in the unconditional case $s=\emptyset$, we zero out the entries of $s$. These embeddings are then concatenated and fed into each block of the UNet.

Training.   During the training stage, as shown in Fig. 2 (a) and (b), the pre-trained style conditioner extracts style features $\{S_{i}\}_{i=1}^{n^{s}}$ for the training instances $\{\mathbf{X_{i}}\}_{i=1}^{n^{s}}$. Each data instance $\mathbf{X_{i}}$, paired with its style $S_{i}$ and a randomly sampled timestep $t\sim\mathcal{U}$, forms a tripartite input $(\mathbf{X_{i}},t,S_{i})$. This setup facilitates the optimization of the model against a loss function defined by Eq. (5).

Sampling.   During the sampling stage, we construct the style combination set $\mathcal{D}$ by Eq. (7). As shown in Fig. 2 (c), a specific style combination $\mathcal{D}_{j}\in\mathcal{D}$ is then selected to guide the diffusion process, generating the new sample that fuses the styles in $\mathcal{D}_{j}$. The sampling operates under single-condition guidance when the style combination comprises a single style, i.e., $|\mathcal{D}_{j}|=1$. Conversely, when the style combination comprises multiple styles, i.e., $|\mathcal{D}_{j}|>1$, the sampling proceeds under multiple-condition guidance.

Domain space diversity.   Through the iterative execution of the sampling procedure, we can generate a diverse range of new, unseen samples that meet domain padding criteria. These synthetic samples collectively form a synthetic dataset $\tilde{D}^{s}$ for TSC models’ training. This process involves two hyperparameters $\kappa$ and $o$. $\kappa$ denotes the proportion of synthetic to original training samples, effectively managing the volume of synthetic samples. $o$ denotes the maximum number of style features that can be combined in each style set.

Training TSC model.   Utilizing the synthetic dataset $\tilde{D}^{s}$, we are able to augment the training dataset to $\{\tilde{D}^{s}\cup D^{s}\}=\{(\mathbf{X}_{i},y_{i})\}_{i=1}^{n^{s}+\tilde{n}^{s}}$. This augmented dataset can be straightforwardly utilized for standard TSC task training. To extract more value from the dataset, we consider the diversity learning strategy from (Qin et al., 2023) for the TSC model’s training. The main thought goes beyond just minimizing not only the standard cross-entropy loss for correct classification. It also involves minimizing the additional cross-entropy loss to effectively differentiate between synthetic and original samples. For more details, please refer to Appendix C and Appendix D.

Datasets.   We assess our method on three widely used HAR datasets: UCI Daily and Sports Dataset (DSADS) (Barshan and Yüksek, 2014), PAMAP2 dataset (Reiss and Stricker, 2012) and USC-HAD dataset (Zhang and Sawchuk, 2012). We follow the same experimental settings in (Qin et al., 2023) that provided a generalizable cross-person scenario. Specifically, the subjects are organized into separate groups for leave-one-out validation. We assign the data of one group as the target domain and utilize the remaining subjects’ data as the source domain. Each subject is treated as an independent task.

Baselines.   We compare our approach with a wide range of closely related, strong baselines adapted to TSC tasks. We first select Mixup (Xu et al., 2020c), RSC (Huang et al., 2020a), SimCLR (Chen et al., 2020), Fish (Shi et al., 2021), and DDLearn (Qin et al., 2023), given their outstanding performance in most recent study (Qin et al., 2023). Notably, DDLearn (Qin et al., 2023) is ranked as the top-performing method. We also include TS-TCC (Eldele et al., 2021) for its remarkable generalization performance in self-supervised learning. Additionally, we incorporate DANN (Ganin et al., 2016) and mDSDI (Bui et al., 2021), which are designed to address domain-invariant and domain-specific feature learning, respectively. In our analysis, the standard data augmentation (DA) techniques (Um et al., 2017) are identical to those employed in (Qin et al., 2023), such as scaling and jittering.

Architecture and implementation. For fairness, we adopt the same feature extractor as described in (Qin et al., 2023), which consists of two blocks for DSADS and PAMAP2, and three blocks for USC-HAD. Each block includes a convolution layer, a pooling layer, and a batch normalization layer. All baselines, except TS-TCC (Eldele et al., 2021), employ this feature extractor. In each experiment, we report the average performance and standard deviation over three random seeds. For detailed experimental setups, including dataset details and training procedures, please refer to Appendix E.

In this part, we demonstrate whether our approach can effectively diversify the domain space and generate diverse samples that meet domain padding criteria. To this end, we adopt T-SNE (Van der Maaten and Hinton, 2008) to visualize the latent feature space in terms of class and domain dimensions.

(1) Class-Preserved Generation. Firstly, we evaluate the class consistency of synthetic data, i.e., the first criterion of domain padding. We employ a class feature extractor, trained with class labels, to map both original and synthetic data into a class-specific latent space. The results of single-condition guidance ($|\mathcal{D}_{j}|=1$) and multiple-condition guidance ($|\mathcal{D}_{j}|>1$) are shown in Fig. 3.

It can be observed that all synthetic samples (crosses) are closely clustered around their corresponding original instances and classes (dots). This clustering indicates that our method effectively maintains class information, avoiding the introduction of class noise; importantly, this holds true under both single and multiple-condition guidance. Moreover, the use of multiple-condition guidances appears to enhance class discriminability more than single-condition guidance in Fig. 3. This enhancement is likely because more guidance signals provide more robust class semantics (akin to ensemble learning), therefore resulting in a better class alignment.

(2) Intra- and Inter-Domain Diversity. Next, we evaluate the intra- and inter-domain diversity of the synthetic data, i.e., the second criterion of domain padding. We train the domain feature extractor on source domains with domain labels. We then map source and target data into a domain-specific latent space, and compare the synthetic data from the standard DA method, single-condition guidance ($|\mathcal{D}_{j}|=1$), and multiple-condition guidance ($|\mathcal{D}_{j}|>1$). The results are shown in Fig. 4.

The findings reveal that the standard DA method generates tightly clustered samples (crosses) around the original data (dots), falling short of diversifying the domain space, particularly the inter-domain space. Our single-condition guidance method offers a partial solution and generates sparse data between different domains thanks to diffusion’s probabilistic nature. However, relying solely on a single-style guidance approach has limitations for domain padding. The introduction of our style combinations in Eq. (7) makes a substantial improvement: the multiple-condition guidance excels in “padding” the distributional gaps both within and between source domains, as demonstrated in Fig.  4.

Moreover, as indicated in Fig. 4, through multiple-condition guidance, the synthetic samples (crosses) closely resemble the target domain data (red dots) and demonstrate less dependence on the specific characteristics of source domains. This demonstrates that the fusion of multiple style features creates a new style. This is of importance for the domain generalization in TSC. Given the significant differences in individual styles and the small-scale nature of source domain data, our style-fused sampling demonstrates great potential to simulate various new and unseen distributions, from which the TSC model can better adapt to the target domains.

In addition, we can observe in Fig. 4 (c) that the USC-HAD dataset presents an additional challenge of intra-domain gaps due to its fragmented and sparsely distributed source domains with distinct sub-domains. These gaps contribute to an increased distribution shift, posing difficulties for existing DG methods to perform effectively (We show their results in Tab. 1 in later). Through random instance-level style fusion, our approach effectively addresses this sub-domain challenge, enabling the synthesis of new data distribution within sub-domains. As a result, our method can yield exceptional performance on the complex tasks like USC-HAD.

Now we conduct a series of experiments to evaluate the generalization performance of DI2SDiff against other strong DG baselines.

Overall performance. Tab. 1 presents a comparative analysis of the classification accuracies achieved by all DG methods across three datasets, each task of which comprises 20% of the training data. As we can see, representation learning baselines that focus solely on learning domain-invariant features, such as DANN (Ganin et al., 2016), exhibit suboptimal performance due to the limited diversity of the training data in HAR. The method mDSDI (Bui et al., 2021), on the other hand, achieves improved performance by additionally learning domain-specific features. However, it does not match the performance of DDLearn (Qin et al., 2023), which utilizes data augmentation, underscoring the importance of training data diversity in enhancing generalization in HAR. In contrast, our DI2SDiff, leveraging advanced synthesis of highly diverse data across both intra- and inter-domain space, markedly surpasses all baseline methods in every task. In addition, we observe that all baselines, including DDLearn, demonstrate poor performance on the USC-HAD dataset. As we discussed in Fig. 4 (c), this decline is due to the presence of sub-domains within the source domain, which poses a highly challenging problem in DG. Nevertheless, DI2SDiff adeptly addresses this issue by integrating instance-level style fusion, thereby synthesizing new data distributions between the sub-domains. As a result, our approach achieves outstanding performance, outperforming the second-best method by a clear margin (6.64%) in USC-HAD.

Data proportion analysis. In Tab. 2, we assess DI2SDiff’s performance over a range of data volumes by adjusting the proportion of training data from 20% to 100%. The results demonstrate DI2SDiff’s consistent superiority over the baseline methods across various proportions of training data. This highlights the ability of our approach to efficiently generate informative samples from varying amounts of available data and effectively learn from them. As the size of the training sample increases, the advantage of our method becomes more pronounced. For instance, as we increase the size from 20% to 100% of USC-HAD, the accuracy improvement grows from 6.64% to 8.62% compared to the second-best baseline (DDLearn). This is because the number of style combinations increases exponentially ($2^{k}-1$) with larger training data volumes, as shown in Eq. (7). Hence, enlarging the training dataset can provide significant diversity enhancement of data synthesis, leading to more substantial gains in the model’s generalization ability.

In this section, we perform an ablation study that focuses on the main step of DI2SDiff, i.e., generating diverse time-series data via diffusion for data augmentation. We keep the number of synthetic samples and the training strategy of TSC models the same for all variants. Additionally, we conduct a sensitivity analysis that focuses on its two hyperparameters: $o$, which controls the maximum number of style features in each style combination, and $\kappa$, which controls the volume of synthetic data.

Ablation on diffusion model.   Our findings, as presented in Table 3, indicate that the standard DA method and class label guidance²²2This involves directly using the class labels, rather than the style features, as the condition to guide diffusion. falter in performance. The failure of the class label guidance sampling suggests that using class labels alone, without instance-related information, cannot generate high-quality data. In contrast, the diffusion model that utilizes a single style feature as a condition achieves better performance, suggesting that leveraging representation features for instance-specific sampling can boost the quality of generated data. Moreover, the incorporation of style-fused sampling can further improve generalization by producing samples with distinct features.

Hyperparameter sensitive analysis. We analyze the sensitivity of our hyperparameters by varying one parameter while maintaining the others constant. As shown in Fig. 5, increasing the complexity of style combinations ($o$) and the volume of synthetic data ($\kappa$) generally leads to performance improvement. It becomes non-sensitive when the value is too large. We find that an $o$ value of 5 for the DSADS and PAMAP2 and an $o$ value of 10 for the USC-HAD sufficiently ensure a diverse range of styles. $\kappa$ values of 1 or 2 strike an effective balance between accuracy and training overhead for all three datasets. By flexibly tuning these hyperparameters, we can achieve even greater performance improvements for the TSC model across various tasks while meeting specific needs.

We demonstrate the versatility of our approach in boosting the performance of existing DG baselines. The results are shown in Fig. 6. By incorporating our synthetic data into the training datasets of baselines, we consistently observe performance improvements across the board, including DANN (Ganin et al., 2016), mDSDI (Bui et al., 2021)³³3In mDSDI, our synthetic data is treated as a new domain., and DDLearn (Qin et al., 2023)⁴⁴4In DDLearn, our synthetic data is treated as a new augmentation method.. This demonstrates the versatility of integrating our method to provide additional gains, making it a practical solution for immediate application. The diverse synthetic data of DI2SDiff is thus ready for use, offering a straightforward way to bolster various baselines without necessitating further data generation.