LLM-based Knowledge Pruning for Time Series Data Analytics on Edge-computing Devices

Our knowledge pruning consists of three steps: knowledge prompt set generation, knowledge anchor point production and pertinent knowledge distillation.

Knowledge Prompt Set Generation Knowledge prompt set (KPS) contains the prompts which are forwarded into a pre-trained LLM for getting the knoweldge anchor points (KAPs). In this paper, these prompts in KPS indicate the description of corresponding data. Since we devise to apply our proposed KP to two fundumental tasks: regression and classification, two different prompt templates are proposed. In regression, the remaining sueful prediction is used to evaluate the performance of our KP, and the prompt template is “The remaining useful life is {num}.”, where num indicates the correspoding groud truth value and ranges from [$y_{min}$, $y_{max}$] In classification, we apply our KP on the human activity recognition, and the prompt template is “The subject is {action}.”, where action means the name of the corresponding activity.

Knowledge Anchor Point Production After obtaining the KPS, we forward these prompts in the KPS to a pre-trained LLM to obtain the language embeddings, which can be formulated as following.

$$z_{i}=F_{l}(\mathcal{P}_{i}),$$

(1)

where $\mathcal{P}_{i}$ indicates the $i$th prompt, $F_{l}$ represents a pre-trained LLM, and $z_{i}$ is the produced language embedding termed as a knowledge anchor point (KAP). These KAPs are used to represent the space of pertinent knowledge.

Pertinent Knowledge Distillation Without transfering the entire knowledge of a LLM, our KP only transfer the pertinent knowledge which is indicated by KAPs. However, there is domain gap between the knowledge learned by LLM and the knowledge of downstream tasks. To alleviate this issue, an alignment module consisting of 2 fully connected layers are used to project these KAPs into tha latent space of the downstream task. This process is computed as below,

$$k_{i}=\phi(z_{i}),$$

(2)

where $\phi$ means the alignment module and $k_{i}$ is the transformed feature vector, which serves as a prior knowledge. Given a segment of time series data $x_{i}$, we forword it into our target model $F_{t}$, and get the predicted feature vector $f_{i}$. To utilize the prior knowledge $\mathcal{Z}=\{z_{i}|i=1,\dots,N\}$, the metric learning is leveraged.

Moreover, to optimze the target model and the alignment simultaneously, based on the unidirectional metric learning in Prototypical Networks Snell et al. (2017), we develope a bi-directional metric learning. For optimizing the target model, the process is computed as:

$$p_{t}(i)=\frac{\exp(-d(k_{i},x_{i}))}{\sum_{t=1}^{|\mathcal{Z}|}\exp(-d(k_{t},x_{i}))},$$

(3)

where $d$ denotes the distance function. To improve the numerical stability and computational efficiency, we further improve this computation progress and compute the prediction as below:

$$p_{t}(i)=\log(\frac{\exp({\rm simi}(k_{i},f_{i}))}{\sum_{t=1}^{|\mathcal{Z}|}\exp({\rm simi}(k_{t},f_{i}))}),$$

(4)

where simi is the cosine similarity. For the alignment optimzation part, the process can be formulated as:

$$p_{l}(i)=\log(\frac{\exp({\rm simi}(k_{i},f_{i}))}{\sum_{t=1}^{|\mathcal{B}|}\exp({\rm simi}(k_{i},f_{t}))}),$$

(5)

where $|B|$ denotes the batch number. Finally, to distill the pertinent knowledge to the target model, the Kullback–Leibler divergence (KL-div) is used and the final loss is:

$$L=0.5*D_{KL}(p_{t},p_{g})+0.5*D_{KL}(p_{l},p_{g}^{T}),$$

(6)

where $p_{g}$ is the ground truth distribution and is defined as following,

$$p_{g}(i)=\frac{\exp(g_{i}*\tau)}{\sum_{t=1}^{|\mathcal{B}|}\exp(g_{t}*\tau))},$$

(7)

where $g_{i}$ is equal to one for correpsonding prompt description, while is zero. Similarly to knowledge distillation, $\tau$ is a temperature hyper-parameter.

Since our KP is based on metric learning, the prediction of the target model is discrete. To extend our KP to the task with continuous output like regression, an anchor voting scheme (AVS) is proposed.

Given the prediction distribution $\mathcal{S}=\{p_{t}(i)|i=1,\dots,|\mathcal{Z}|\}$, we firstly sort these scores in a descending order as below:

$$\hat{\mathcal{S}}={\rm sort}(\mathcal{S}).$$

(8)

After that, these scores are cumulated according to Eq. 9.

$$\mathcal{S}_{a}={\rm cumsum}(\hat{\mathcal{S}})$$

(9)

Then, the cumlated scores which are larger than $\theta$, are formed into a voting set $\mathcal{V}=\{v_{i}|1,\dots,|\mathcal{V}|\}$. The final prediciton is generated as following,

$$o=\frac{\sum_{i=1}^{|\mathcal{V}|}v_{i}*n_{i}}{\sum_{i=1}^{|\mathcal{V}|}v_{i}},$$

(10)

where $n_{i}$ indicates the numerical value described by the KAP $v_{i}$. With our proposed AVS, our proposed KP is effectively entended to the regression task, achieving significant performances.Chen et al. (2020)

Datasets To comprehensively investigate the performances of our KP, two fundamental tasks on edge-computing devices: classification and regression, are evaluated in this paper. In classificaiton, the human activity recognition (HAR) task is studied and four different benchmarks: UCI_HAR Anguita et al. (2013), Opportunity Roggen et al. (2010), PAMAP2 Reiss and Stricker (2012), and WISDM Kwapisz et al. (2011) are used. These benchmarks contain different number of activity categories ranging from 6 to 17 with different scales between 3k and 29k samples. In regression, the remainning useful life (RUL) prediciton is alleviated for evaluation, where the C-MAPSS Saxena et al. (2008) dataset is used. C-MAPSS contains four different subsets: FD001, FD002, FD003 and FD004 with different scenarioes.

Experimental Setup In classifiction, for consistency and meaning fulcomparison, the training and inference process on UCI_HAR, Opportunity and PAMAP2 are conducted according to the protocol of iSPLInception Ronald et al. (2021). Since the experiments in iSPLInception Ronald et al. (2021) do not include WISDM benchmark, the expriments on WISDM follow the setting in Multi CNN-BiLSTM Challa et al. (2022). For fair comparion, other compared methods are re-implemented under the same setting. According to approches Ronald et al. (2021); Challa et al. (2022), F1-Score is used for evaluation in HAR tasks. In regression, some methods are also re-implemented under the same conditions. The training and inference processes are conducted according to classic RUL methods Jin et al. (2022a); Chen et al. (2020). RMSE and scoring fuction are used as evaluation metrics. Two hyper-parameters $\tau$ and $\theta$ are set as 10 and 0.9, respectively for all experiments. The pre-trained text encoder in CLIP Radford et al. (2021) is used as the pre-trained LLM in experiments. Experiments are conducted on a workstation with a GeForce RTX 4080 GPU and 128 GB memory from 1 to 4 hours.

To evaluate the performances of our KP, we compare our approach with orther state-of-the-art (SOTA) methods. The exprimental results in regression and classificationare listed in Table. 1 and Table. 2, respectively.

In the RUL task, serveral SOTA approaches: Li et al. Li et al. (2018), BLCNN Liu et al. (2019), PE-Net Jin et al. (2022b), DGRU Behera and Misra (2021), AdaNet Jin et al. (2023d), Jang et al. Jang and Kim (2021) and KDnet Xu et al. (2021) , are compared with our method. Among these methods, Li et al. proposes a CNN based network to predict the RUL. PE-Net integrates position encoding scheme with an optimzed CNN architecture for the RUL task. AdaNet introduces the deformable convolution into the RUL task. BLCNN devises a hybrid network which combines RNN and CNN together to improve the prediction accuracy. DGRU apply the adversarial learning on the RUL task. A self-supervised learning approch is proposed in Jang et al. KDnet utilize knowledge distillation to transfer the knowledge in RNN to a CNN model. Benefitted from the learned pertinent knowledge from a pre-trained LLM, our KP performs much better than them and achieve the best performances.

In the HAR task, we compare our KP with seven SOTA methods: LSTM-CNN Xia et al. (2020), CNN Van Kuppevelt et al. (2020), Multi CNN-GRU Dua et al. (2021), Multi CNN-BiLSTM Challa et al. (2022), GRU_INC Mim et al. (2023), DTL Ige and Noor (2023) and iSPLInception Ronald et al. (2021). These compared approaches employ different network architectures like RNN, CNN and even hybrid networks. As a novel comdel compression paradigm, our proposed KP is fundamentally orthogonal to existing HAR methods and can be applied to any existing HAR approaches. We apply our KP to two different methods: DTL and iSPLInception, and list the best performances we achieved in Table 1. Through experiments, it demonstrates that our KP effectively transfer the pertinent knowledge of a pre-trained LLM to the target model, which achieves the best performances among other SOTA methods.

To verify the effectiveness, experiments on ablation study are presented. Our KP is orthogonal to approaches for time series analytics and can be directly applied to these methods. To show the generalization of our KP, we apply our KP on serveral different networks and show the performance improvements. Experimental results on the regression task, RUL, and classification task, HAR are listed in Table. 3 and Table. 5, respectively.

In the RUL task, three different approaches: Bi-LSTM, Two-Stream BiLSTM Jin et al. (2022a) and PE-Net Jin et al. (2022b) are used as our baselines. Bi-LSTM is a shallow network, which consists of two bi-directional LSTM. Two-Stream BiLSTM integrates the handcrafted feature flow Jin et al. (2022a) into the raw time series data via a Bi-LSTM based network. PE-Net designs a CNN with a position encoding scheme to predict the RUL. In Table. 5, Bi-LSTM is used as baseline1, PE-Net is used as baseline2 and Two-Stream Bi-LSTM is used as baseline3. With our proosed KP, all these three methods are remarkablely improved among four different scenarioes. Compared with RMSE, Score is generally regarded as a more important evaluation metric, since it give more penalty on the late prediciton, which is similar to the practical setting. Among these three baselines, baseline3 acheves the best performances on average. After applying our KP, its performances are further improved by 19.7% in Score.

In the HAR task, two different methods: DTL Ige and Noor (2023) and iSPLInception Ronald et al. (2021), are used as our baselines. DTL is a hybrid network, which combines CNN and RNN togethor to capture the temporal features. In comparison, iSPLInception proposes to utilize the inception based CNN network to classify the human activaities.In Table. 3, baseline1 indicates the DTL, and basleine2 represents the iSPLInception. According to the experimental results, our KP is able to effectively improve the performances on these two baselines among all four benchmarks. The improvements on these methods range from 0.8% to 13.7%.

These expriments above show that our propsoed KP effectively identify the pertinent knowledge and transfer it to the target model. With our KP, all these five baselines are improved by a large marge.

Effectiveness of AVS AVS is proposed to enable the metric learning based network to predict continuous value for the regression task, like RUL. Experiments are designed to show the effectiveness of our proposed AVS, which are listed in Table 4. The Bi-LSTM is used as the baseline. The experimental reults indicates that without our propsoed AVS, the performances of KP on the regrression task are not satisfactory. After applying our proposed AVS, our KP can effectively improve the accuracy on the RUL prediction by a large marge.

Based on the experiments above, it can be found that our proposed KP can consistently improve the performances across different tasks and benchmarks. Since the improvemment by KP ranges from 0.8% to 13.7%, the effectiveness of our KP may be affected by the specific data distribution and the neural network architecture.

Our KP is proposed to alleviate the issue of the computation cost in LLMs. The experiments on computation efficiency is carries out and listed in Table. 6, where the FLOPs and Params are listed to compare the computation efficiency.

As listed in Table. 6, we apply our KP to two networks: Two-Stream BiLSTM and DTL for the task RUL and HAR, respectively. Since DTL applies a hybrid network, which is composed of CNN and RNN and is more complex than the Two-Stream BiLSTM, the computation complexity of DTL is higher than that of Two-Stream BiLSTM. Nevertheless, the computation demands on these two approaches are much lower than that of the LLM in CLIP. According to the experimental results, our proposed KP is able to effectively prune the redundant knowedlge of LLM. The computation issue of LLMs is well alleviated, and the performances of the target model are improved.

Our proposed KP involves two hyper-parameters: $\tau$ and $\theta$. To investigate the impact of different values of these two parameters, several experiments are conducted and discussed. For the parameter $\tau$, we graduately increase its values and carry out experiments on HAR and RUL tasks, respectively. These experimental results on RUL and HAR are illurstrated in Fig. 3 and Fig. 4, respectively.

In Fig. 3, our KP is applied on Two-Stream BiLSTM with different $\tau$ values. Experiments are carried out in FD004 subset, which contains the most complex scenarioes. Although the difference performances are obtained on RUL task, their performances are still better than the baseline. With our KP, the performances of Two-Stream BiLSTM are consistly improved under different values of $\tau$.

In Fig. 4, we apply our KP on the DTL in two different datasets: UCI_HAR and WISDM benchmarks. It shows that our KP can improves the performances of DTL under different $\tau$ values.

AVS is proposed to regression tasks, which enables our network to predict continuous value for the RUL task. The hyper-parameter $\theta$ in AVS is used as a threshold value to select the anchor for voting. To investigate the stability of our AVS, we apply our KP to the Two-Stream BiLSTM and design expriments on FD004 with different $\theta$, which are presented in Table. 7.

As listed in Table 7, our AVS with different $\theta$ values is able to consistly improve the performances of Two-Stream BiLSTM. This demonstrates that our proposed AVS is adaptive to the variation of $\theta$.