Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement

Unsupervised Domain Adaptation (UDA) for time-series data addresses the critical challenge of distribution shifts between the source and target domains. Discrepancy-based methods, such as those of Cai et al. (2021), Liu & Xue (2021) and He et al. (2023), align feature representations through statistical measures, while adversarial approaches such as Wilson et al. (2020), Wilson et al. (2023), Ragab et al. (2022), Jin et al. (2022) and Ozyurt et al. (2023) focus on minimizing distributional discrepancies through adversarial training. The comprehensive study by Ragab et al. (2023a) provides a broad overview of domain adaptation in time series. However, SFDA assumes access to only the source-pretrained model (Liang et al., 2020; 2021; Li et al., 2020; Yang et al., 2021a; 2022; 2023; Kim et al., 2021; Xia et al., 2021). In SFDA, one of the most commonly used strategies is based on unsupervised clustering techniques (Yang et al., 2022; Li et al., 2024), promoting the discriminability and diversity of feature spaces through information maximization (Liang et al., 2020), neighborhood clustering (Yang et al., 2021a) or contrastive learning objectives (Yang et al., 2022). Recent advances incorporate auxiliary tasks, such as masking and imputation, to enhance SFDA performance, as demonstrated by Ragab et al. (2023b) and Gong et al. (2024), taking inspiration from Liang et al. (2021) and Kundu et al. (2022a) to include masked reconstruction as an auxiliary task. However, exploration of SFDA in time series contexts remains limited, especially with regard to parameter and sample efficiency during target-side adaptation (Ragab et al., 2023a; Gong et al., 2024).

Neural network pruning has demonstrated that substantial reductions in parameters, often exceeding 90%, can be achieved with minimal accuracy loss, revealing significant redundancy in trained deep networks (LeCun et al., 1989; Hassibi & Stork, 1992; Li et al., 2017; Frankle & Carbin, 2018; Sharma et al., 2024). Structured pruning methods, such as those of Molchanov et al. (2017) and Hoefler et al. (2021), have optimized these reductions to maintain or even improve inference speeds. Low-rank models have played a crucial role in pruning, with techniques such as Singular Value Decomposition (SVD) (Eckart & Young, 1936) applied to fully connected layers (Denil et al., 2013) and tensor-train decomposition used to compress neural networks (Novikov et al., 2015). Methods developed by Jaderberg et al. (2014), Denton et al. (2014), Tai et al. (2016), and Lebedev et al. (2015) accelerate Convolutional Neural Networks (CNNs) through low-rank regularization. Decomposition models such as CANDECOMP/PARAFAC (CP) (Carroll & Chang, 1970) and Tucker decomposition (Tucker, 1966) have effectively reduced the computational complexity of CNNs (Lebedev et al., 2015; Kim et al., 2016). Recent works have extended these techniques to recurrent and transformer layers (Ye et al., 2018; Ma et al., 2019), broadening their applicability. In Figure 2A, we identify significant parameter redundancies in source-pretrained models in terms of effective parameter ranks, revealing low effective ranks. Our motivation for a factored re-parameterization is driven by the specific demands of our proposition—parameter and sample efficiency. We utilize low-rank tensor factorization (Tucker-style decomposition) to disentangle weight tensors into compact disentangled subspaces, enabling selective fine-tuning during target adaptation. This strategy enhances the parameter efficiency of the adaptation process and implicitly mitigates overfitting. As shown in Figure 2B, the validation F1 score for baseline methods declines over epochs, whereas our proposed SFT method maintains consistent performance, demonstrating its robustness against overfitting.

After the supervised source pre-training (cf. Appendix A.7 for details on source pre-training), we adopt Tucker-style factorization (Tucker, 1966; Lathauwer et al., 2000) of the weight tensors for its effectiveness in capturing multi-way interactions among different mode-independent factors. The compact core tensor encapsulates these interactions, and the factor matrices in the decomposition offer low-dimensional representations for each tensor mode. As illustrated in Figure 3A, the rank-$(R_{1},R_{2},R_{3})$ Tucker-style factorization of the 3-way tensor $\boldsymbol{\mathcal{A}}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}}$ is represented as:

where $\boldsymbol{\mathcal{G}}\in\mathbb{R}^{R_{1}\times R_{2}\times R_{3}}$ is referred to as the core tensor, and $\mathbf{U}^{(1)}\in\mathbb{R}^{I_{1}\times R_{1}}$, $\mathbf{U}^{(2)}\in\mathbb{R}^{I_{2}\times R_{2}}$ and $\mathbf{U}^{(3)}\in\mathbb{R}^{I_{3}\times R_{3}}$ as factor matrices.

Furthermore, the linear operations involved in Tucker-style representation (Equation (1)), which are primarily mode-independent matrix multiplications between the core tensor and factor matrices, simplify both mathematical treatment and computational implementation. This linearity ensures that linear operations (e.g., convolution or linear projection) in deep networks using the complete tensor can be efficiently represented as sequences of linear suboperations on the core tensor and factor matrices.

For instance, in one-dimensional convolutional neural networks (1D-CNNs), the convolution operation transforms an input representation $I\in\mathbb{R}^{C_{\text{in}}\times L}$ into an output representation $O\in\mathbb{R}^{C_{\text{out}}\times L^{\prime}}$ using a kernel $\boldsymbol{\mathcal{W}}\in\mathbb{R}^{C_{\text{out}}\times C_{\text{in}}\times K}$. Here, $C_{\text{in}}$, $C_{\text{out}}$, $K$, $L$, and $L^{\prime}$ denote the number of input channels, output channels, kernel size, input sequence length, and output sequence length, respectively. The operation is mathematically defined as:

Given that the dimensions of the channels ($C_{\text{in}}$ and $C_{\text{out}}$) are typically much larger than the size of the kernel ($K$), we restrict the decomposition to modes 1 and 2 only, focusing on the input and output channels. The 2-mode decomposition is represented by:

where $\boldsymbol{\mathcal{T}}\in\mathbb{R}^{R_{\text{out}}\times R_{\text{in}}\times K}$ is the 2-mode decomposed core tensor, and $\mathbf{U}^{(1)}\in\mathbb{R}^{C_{\text{out}}\times R_{\text{out}}}$ and $\mathbf{U}^{(2)}\in\mathbb{R}^{C_{\text{in}}\times R_{\text{in}}}$ represent the respective factor matrices, also illustrated in Figure 3B. With this decomposed form, the convolution operation can be represented as a sequence of the following linear operations:

where both the Channel Down-Projection and the Channel Up-Projection operations are implemented as unit window-sized convolution operations.

Building on these insights, we decompose the pre-trained source model weights using Tucker decomposition, optimized via the Higher-Order Orthogonal Iteration (HOOI) algorithm, an alternating least squares method for low-rank tensor approximation (detailed algorithm in Appendix A.1). The optimization problem is defined as:

The weight tensor is then re-parameterized with the core tensor $\boldsymbol{\mathcal{T}}^{*}\in\mathbb{R}^{R_{\text{out}}\times R_{\text{in}}\times K}$ and mode factor matrices $\mathbf{V}^{(1)*}\in\mathbb{R}^{C_{\text{out}}\times R_{\text{out}}}$ and $\mathbf{V}^{(2)*}\in\mathbb{R}^{C_{\text{in}}\times R_{\text{in}}}$, as formulated in Equations (4), (5), and (6).

However, the re-parameterization is performed after minimizing the linear reconstruction error of the weights (Equation 7), which may degrade the predictive performance on the source domain (Kim et al., 2016). To mitigate this effect, we fine-tune the core tensor and factor matrices using the source data for an additional 2-3 epochs, prior to deploying the model on the target side. This fine-tuning effectively restores the original predictive performance, preserving the model’s performance while leveraging the benefits of fewer parameters. This behavior strongly aligns with the Lottery Ticket Hypothesis (Frankle & Carbin, 2018), which suggests that compressing and retraining pre-trained, over-parameterized deep networks can result in superior performance and storage efficiency compared to training smaller networks from scratch. Similar observations have been made by Zimmer et al. (2023) in the context of large language model training.

Moreover, we introduce the rank factor ($RF$), a hyperparameter to standardize the mode rank analysis. The mode ranks are set as $(R_{\text{in}},R_{\text{out}})=\left(\left\lfloor\frac{C_{\text{in}}}{RF}\right\rfloor,\left\lfloor\frac{C_{\text{out}}}{RF}\right\rfloor\right)$, where a higher $RF$ results in lower mode ranks, and vice versa. Although $RF$ can be set independently for the input and output channels, for simplicity in our analysis, we control both $R_{\text{in}}$ and $R_{\text{out}}$ with a single hyperparameter, $RF$. This tunable parameter allows for flexible control over the trade-off between parameter reduction and model capacity.

The core tensor in our setup plays a pivotal role by capturing multi-way interactions between different modes, encoding essential inter-modal relationships with the factor matrices. Additionally, its sensitivity to temporal dynamics (cf. Equation (5)) makes it particularly well-suited for addressing distributional shifts in time-series data, where discrepancies often arise due to temporal variations (Fan et al., 2023). Therefore, by selectively fine-tuning the core tensor during target adaptation, we can effectively adapt to these shifts while maintaining a compact parameterization. Figure 4 presents a toy experiment that demonstrates how fine-tuning the core tensor suffices in addressing domain shift, aligning the model more effectively with the target domain. This approach not only ensures that the model aligns better with the target domain but also enhances parameter efficiency by reducing the number of parameters that need to be updated, minimizing computational overhead. Moreover, selective fine-tuning mitigates the risk of overfitting, providing a more robust adaptation process, as shown in Figure 2B. This strategy leverages the expressiveness of the core tensor to address temporal domain shifts while maintaining the computational benefits of a structured, low-rank parameterization.

The decomposition significantly reduces the parameter count from $C_{\text{out}}\times C_{\text{in}}\times K$ to:

where $R_{\text{out}}=\left\lfloor\frac{C_{\text{out}}}{RF}\right\rfloor\ll C_{\text{out}}$ and $R_{\text{in}}=\left\lfloor\frac{C_{\text{in}}}{RF}\right\rfloor\ll C_{\text{in}}$, ensuring substantial parameter savings. Furthermore, since we selectively finetune the core tensor, the parameter efficiency is further enhanced.

In terms of computational efficiency, the factorized convolution reduces the operation count from $\mathcal{O}(C_{\text{out}}\times C_{\text{in}}\times K\times L^{\prime})$ to a series of smaller convolutions with complexity:

resulting in a notable reduction in computational cost, dependent on the values of $R_{\text{out}}$ and $R_{\text{in}}$ (see Appendix A.2 for a detailed explanation).

In Equation (10), the test error (LHS) is bounded by the empirical training loss and the divergence between the source pre-trained weights ($\boldsymbol{W}_{\text{src}}$) and the target fine-tuned weights ($\boldsymbol{W}_{\text{trg}}$), measured using the layer-wise Frobenius norm. Motivated by this theoretical insight, we empirically analyze the layer-wise parameter distances between the source and target models in terms of the Frobenius norm. Figure 5 illustrates these distances for both vanilla fine-tuning (Baseline: Fine-tuning all parameters) and our SFT approach which selectively fine-tunes the core subspace, across various time series datasets, including SSC (Goldberger et al., 2000), MFD (Lessmeier et al., 2016), and HHAR (Stisen et al., 2015), for different $RF$ values. The results show that SFT naturally constrains the parametric distance between the source and target weights, with a higher $RF$ exhibiting a greater constraining ability, thus regulating the adaptation process in line with the distance-based regularization hypothesis of Gouk et al. (2021). Furthermore, in Appendix A.3 (Lemma 2), we provide a formal bound on the parameter distance in terms of the decomposed ranks of the weight matrices. This observation reveals a trade-off in the source-target parameter distance; while a smaller distance tightens the generalization bound, it also limits the model’s capacity, potentially regularizing or, in extreme cases, hindering adaptability. In Section 4, we use this bound to further discuss SFT’s sample efficiency.

We utilize the AdaTime benchmarks proposed by Ragab et al. (2023a; b) to evaluate the SFDA methods: SSC (Goldberger et al., 2000), and MFD (Lessmeier et al., 2016), HHAR (Stisen et al., 2015), UCIHAR (Anguita et al., 2013), WISDM (Kwapisz et al., 2011). Here each dataset involves distinct domains based on individual subjects (SSC), devices (HHAR, UCIHAR, WISDM) or entities (MFD). For comprehensive dataset descriptions and domain details, refer to the Appendix A.5. To assess the effectiveness and generalizability of our decomposition framework, we integrate it with prominent SFDA methods: SHOT (Liang et al., 2020), NRC (Yang et al., 2021a), AAD (Yang et al., 2022), and MAPU (Ragab et al., 2023b) within time-series contexts. For more details on the adaptation method we direct readers to Appendix 8.

Our experimental setup systematically evaluates the proposed strategy (SFT) alongside contemporary SFDA methods evaluated by Ragab et al. (2023b); Gong et al. (2024), including SHOT (Liang et al., 2020), NRC (Yang et al., 2021a), AAD (Yang et al., 2022), and MAPU (Ragab et al., 2023b). In SFT, the backbone fine-tuning is restricted to the core subspace of the decomposed parameters, and the hyperparameters for source training and target adaptation are kept consistent between each vanilla SFDA method and its SFT variant. In addition, we assess the impact of different ranking factors ($RF$). For values of $RF$ greater than $8$, we observed a significant underfitting, leading us to limit our evaluation to $RF\in\{2,4,8\}$. The experiments are carried out using the predefined source-target pairs from the AdaTime benchmark (Ragab et al., 2023a). The predictive performance of the methods is evaluated by comparing the average F1 score of the target-adapted model across all source-target pairs for the respective dataset in the benchmark. Each experiment is repeated three times with different random seeds to ensure robustness. More details are provided in Appendix A.7.

We assess the sample efficiency of the proposed SFT method by evaluating its predictive performance in the low-data regime. To conduct this analysis, we randomly select 0.5% and 5% of the total available unlabeled target samples in a stratified manner, utilizing fixed seeds for consistency. These sampled subsets serve exclusively for the adaptation process. In Figure 6, we present a comparative analysis of the average F1 score post-adaptation, contrasting SFT with baseline methods (SHOT, NRC, AAD, and MAPU) across multiple datasets: SSC, MFD, HHAR, WISDM, and UCIHAR. Notably, our empirical observations are grounded in the theoretical framework established by Theorem 1, which is encapsulated in Equation (10). While all SFDA methods aim to minimize the empirical loss on the target domain (the first term on the right-hand side of Equation (10)) through their unsupervised objectives, the second term—dependent on both the number of samples and the distance between parameters—plays a critical role in generalization. In low-data regimes, as the number of target samples reduces, this second term increases, thereby weakening the generalization bound. As a result, baseline methods exhibit a noticeable drop in predictive performance, distinctly observed for the MFD and UCIHAR datasets (Figure 6), when adapting under the low sample ratio setting. In contrast, SFT effectively mitigates the adverse effects of a small number of samples by implicitly constraining the parameter distance term (the numerator of the second term on the RHS), as empirically observed in Figure 5, therefore, balancing the overall fraction to not loosen the bound on the RHS. This renders SFT significantly more sample-efficient, preserving robust generalization even in low-data regimes. We extend this analysis in Appendix A.7.

In addition, Table 1 demonstrates consistent improvements in the F1 score alongside enhanced parameter efficiency during the adaptation process, as evidenced by the reduced parameter count (# Params.) of the parameters fine-tuned. As the rank factor ($RF$) increases, the size of the core subspace decreases (cf. Equation (8)), leading to fewer parameters that require updates during adaptation. This reduction not only enhances efficiency but also lowers computational costs (MACs) (cf. Equation (9)). The observed performance gains and improved efficiency are consistent across different SFDA methods and datasets, underscoring the robustness, generalizability, and efficiency of our approach across different SFDA methods and benchmark datasets. In Appendix A.7, we extend this analysis to WISDM and UCIHAR datasets.

To further substantiate the robustness of SFT, we benchmark its performance against Parameter-Efficient Fine-Tuning (PEFT) approaches (Han et al., 2024; Xin et al., 2024). PEFT methods primarily aim to match the performance of full model fine-tuning while substantially reducing the number of trainable parameters. Among the most prominent approaches, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has gained significant traction in this domain. In Figure 7, we analyze the performance of LoRA at varying intermediate ranks ({2, 4, 8, 16}), and Low-Rank Kronecker Adaptation (LoKrA) (Edalati et al., 2022; Yeh et al., 2024) under its most parameter-efficient configuration, alongside SFT, on the SSC dataset (additional results are provided in Appendix Figure 11). Remarkably, SFT consistently outperforms across SFDA tasks, demonstrating superior adaptability and performance. This advantage arises from the structured decomposition applied during source model preparation, which explicitly disentangles the parameter subspace. In contrast, LoRA-type methods introduce a residual branch over the pretrained weights that relies on the fine-tuning objective to uncover the underlying low-rank subspaces (Hu et al., 2021), which results in weaker control over the adaptation process. It is important to highlight that PEFT methods can also be seamlessly integrated with SFT during fine-tuning. For instance, LoRA or LoKrA-style adaptation frameworks can be applied to the core tensor, further enhancing the efficiency of the adaptation process, which we discuss in detail in Appendix A.8. However, while LoRA-type methods can effectively adapt model weights to the target domain, they do not reduce inference overhead. Achieving this requires explicit decomposition and reparameterization of the weights.