Large Language Model Aided QoS Prediction for Service Recommendation

Among QoS prediction methods for web services, collaborative filtering (CF) [11] methods receive extensive attention and are widely regarded as the most effective approach. CF-based QoS prediction methods can be divided into two main categories: memory-based methods and model-based methods.

For memory-based CF methods, the similarity of users or services is first calculated. Then QoS values of the target user and service are predicted based on both the historical QoS values and the similarity relationships with their neighbors. Shao et al. [12] first utilize user-based CF methods for QoS prediction. Qi et al. [13] identify “enemies” of the target user to determine their “potential friends”, successfully overcoming the cold-start problem. Meanwhile, item-based methods are widely employed [14, 15]. Chen et al. [16] propose a predictive method that integrates service similarity and geographical location information, effectively alleviating data sparsity and cold-start issues. Additionally, hybrid algorithms that compute both service-to-service and user-to-user similarities [17] are developed. Zheng et al. [18] simultaneously consider user and service similarities for QoS prediction, demonstrating superior predictive performance compared to previous methods. Hu et al. [19] propose to integrate time information to improved CF for QoS prediction.

Model-based CF methods typically use models to predict the QoS values, with matrix factorization (MF) being the most common approach. MF methods decompose the user-service interaction matrix into user and service matrices, predicting values by multiplying these two matrices. Zhu et al. [20] notably extend the traditional MF models by incorporating techniques including data transformation, online learning, and adaptive weighting on the basis of CF techniques. Zheng et al. [21] develop a neighborhood-integrated MF method for personalized web service QoS prediction. Xu et al. [22] employ a probabilistic matrix factorization (PMF) model to learn and predict QoS values, incorporating geographical location to identify user neighbors and comprehensively consider their impact on web service invocation experiences.

Deep learning technology has demonstrated remarkable capabilities across various fields of artificial intelligence in recent years [23, 24, 25, 4, 5]. And an increasing number of deep learning based techniques for QoS prediction are developed [26, 27] too. Zou et al. [28] propose a novel adaptive QoS prediction framework, utilizing a dual-tower deep residual network to extract latent features of users and services and predict QoS values. Wu et al. [29] introduce a reputation-integrated graph convolutional network for robust and accurate QoS prediction. Wei et al. [30] use a transformer based method to learn the graph structure for time-aware service recommendation. Zhu et al. [31] develop a twin graph network based on graph contrastive learning, effectively addressing cold-start and data sparsity issues. Zhou et al. [32] propose the spatial context-aware time series forecasting (SCATSF) framework by utilizing both temporal and spatial contextual information of users and services. Lian et al. [33] facilitate multi-task learning for training QoS prediction model.

Most deep learning based QoS prediction models use the user and service ID information through embedding layers. Additionally, to improve prediction performance, some methods [28, 31] utilize more information of the users and services such as their location, IP address, and the autonomous systems they belong to. Such information provides similarity or proximity signal for the model, which are useful for QoS prediction. In our work, we utilize such information through large language models.

In the field of natural language processing (NLP), LLMs have seen rapid development since the introduction of the transformer architecture [34]. They have shown state-of-the-art performance across virtually all NLP tasks, including text classification and summarizing, machine translation, text generation, and human-like conversation systems [4, 5].

Most LLMs are based on variants of the already ubiquitous transformer architecture, with the core components being the attention based mechanism that can effectively capture the correlation among different words in the text. It is common for a group of researchers to develop a series of LLMs that share similar architecture and training recipes, but with generational improvement in performance and capability. Notable ones include the BERT series of original BERT[6], RoBERTa [7], ALBERT [35], and DeBERTa [7]; the OpenAI GPT series of GPT [36], GPT-2 [37], GPT-3 [8], and GPT-4 [38]; the META LLaMA series of LLaMA [39], LLaMA2 [40], and LLaMA3 [10]; and the Microsoft Phi series of Phi1 [41, 42], Phi2 [43], and Phi3 [9]. The common trend in each series is that the newer models usually have more parameters, and are trained on larger corpus of text. Larger LLMs have higher capacity, can learn more information from a larger corpus of text, have a higher generalization capability, and in general perform better on standardized NLP benchmarks [44].

Thought LLMs are trained from textual data, they can generalize to tasks beyond NLP too. They have been shown to work on visual question-answering [45], open-vocabulary object detection [46], planning [47], and autonomous driving [48]. In this paper, we use LLMs for the service recommendation task. When using LLMs on non-NLP tasks, it is important to convert the data into a form of prompt that can be processed by the LLMs. In our work we construct descriptive sentences from the relative attributes of web users and services to be used by LLMs. More details are discussed in § III-A.

Different LLMs have different levels of accessibility. The parameters of some LLMs can be downloaded and ran on local computers for any end user, which is often referred as “open-weight” models. Some models are only be used through API calls and the model parameters cannot be accessed. In this paper we only use open-weight models for reproducibility. However our method can be applied to any LLMs with feature extraction functionality.

For each web user and service in WSDream dataset, we use the corresponding attributes to construct a descriptive sentence as the input to the LLMs for feature extraction. Specifically, we use the country and autonomous system for users. For instance, the descriptive sentence for one of the user is \asciifamily”web user, located in United States, in autonomous system AS5661 USF - UNIVERSITY OF SOUTH FLORIDA.” Similarly, the URL, service provider, country, and autonomous system are used to services. For example one of the services has the descriptive sentence of \asciifamily”web service, at url http://biomoby.org/services/wsdl/ualberta.ca /DrugBankByName, hosted by ualberta.ca, located in Canada, in autonomous system AS3359 University of Alberta.” We do not include numerical attributes such as IP address, latitude, and longitude in the sentences, as LLMs usually cannot extract meaningful information from numbers. For each user or service, we use the same descriptive sentence for different LLMs.

After obtaining the descriptive sentence for each user or service, we use pre-trained RoBERTa [7] or Phi3mini [9] to extract sentence-level language features. The pipeline is illustrated in Figure II. The sentence is first converted into a sequence of tokens by a tokenizer. The embedding vector of each token is obtained by their token ID and position. A set of stacked LLM layers process on this sequence of embedding vectors to obtain the final sequence of feature vectors. One of the vector in the final sequence is used as the LLM feature vector of the descriptive sentence.

RoBERTa and Phi3mini have similar overall architecture. Each LLM layer in them comparably consists of an attention based layer [34] and a set of feed-forward layers. The attention layer calculates the correlation among the feature vectors in the sequence. The feed-forward layers are combination of linear layers, normalization layers, and activation functions. This common architecture is used by many other LLMs too.

However, RoBERTa and Phi3mini have different number of layers, different dimensionality of the embedding vectors, and different tokenizer. Phi3mini has $3.5\times 10^{9}$ parameters, which is significantly larger than RoBERTa’s $1.2\times 10^{8}$ million parameters. They are also pre-trained on different datasets using different training tasks. RoBERTa is pre-trained to do text classification on total of 160 giga-bytes of text data. If each token uses 5 bytes on average, the model sees $3.2\times 10^{10}$ tokens during training. For text classification, the first vector in the final sequence is used. So we also use the first vector as the sentence-level feature. On the other hand, Phi3mini is pre-trained using next token prediction on a dataset with $3.3\times 10^{12}$ tokens. Next token prediction uses the last vector in the final sequence, so we also use the last vector in the final sequence for Phi3mini. Later we show in experiments that Phi3mini outperforms RoBERTa consistently.

The overall architecture of the proposed large language model aided QoS prediction (llmQoS) model is illustrated in Figure III. First, both the user and service IDs are projected to feature vectors by their corresponding embedding layers. Formally for user $u$ and service $s$,

where $i_{u}$ and $i_{s}$ are the IDs of the user and the service, and $Embed_{u}$ and $Embed_{s}$ are the corresponding embedding layers.

Then descriptive sentences are constructed and the LLM features are extracted for each user and service based on their attributes, as described in § III-A. The descriptive LLM features have much larger dimensionality compared to the ID embedding features. So a projection layer is used for to reduce them to the same dimensionality as the ID embedding features:

where $t_{u}$ and $t_{s}$ are the descriptive sentences of the user and service. Please note that we use the same $LLM$ model and projection layer $Proj$ for both user and service.

Then the user ID embedding feature vector, the projected user descriptive LLM feature vector, the service ID embedding feature vector, and the projected service descriptive LLM feature vector are all concatenated. This concatenated feature vector is used as the input to an multi-layer perceptron (MLP) network. Finally a linear layer produces the predicted QoS value for the input user-service pair. Formally,

where $\hat{y}$ is the predicted QoS value for $u$ and $s$.

Please note that our network design is straightforward without complicated architectures such as graph convolution or attention. The QoS prediction network architecture is not the focus of this paper. With straightforward architecture, llmQoS can still achieve state-of-the-art performance as shown by the experiments. This demonstrate the effectiveness of incorporating LLM features for QoS prediction. In § IV-H we further demonstrate that LLM features can be integrated into more complex QoS prediction models and can still increase the performance significantly and consistently.

To evaluate the performance of our model, we conducted extensive experiments on the WSDream dataset [49]. WSDream is a large-scale real-world dataset of Internet user-service interactions that involves 5,825 services and 339 users in total. Each user and service is identified by a unique ID number. In addition to the IDs, WSDream also provides basic attribute information for each user and service. User attributes consist of IP Address, Country, IP Number, Autonomous System, Latitude, and Longitude. Service attributes include WSDL Address, Service Provider, IP Address, Country, IP Number, Autonomous System, Latitude, and Longitude. We utilize both the IDs and attributes for the users and services in our llmQoS model.

WSDream contains QoS record that include 1,873,850 response time values and 1,831,265 throughput values. For experiments, both the throughput sub-dataset and the response time sub-dataset are randomly split into training and testing sets. We use different density values $D$, which is the percentage of training set data in the total dataset, to test the models’ prediction ability at different data sparsity levels. Specifically we use $D$=5%, 10%, 15%, and 20%. It is expected that with all other settings unchanged, QoS prediction models achieve higher precision with larger values of $D$.

We implemented our model using \asciifamilyKeras 2.0.9 [50] with a \asciifamilyTensorFlow 1.4.0 [51] backend. The ID embedding for both user and service has a dimensionality of 16. Both pre-trained RoBERTa and Phi3mini are taken from Huggingface [52], and the descriptive LLM feature extraction is implemented using the \asciifamilyTransformers library. The descriptive LLM feature vectors from RoBERTa and Phi3mini have 768 and 3,072 dimensions, respectively. In both cases the descriptive LLM feature vector is reduced to 16 dimensions by the projection layer. The MLP module has 3 linear layers with output dimensionalities of 32, 16, and 8. Each linear layer is followed by a ReLU activation function [53]. The final prediction layer is a single linear layer that maps the 8-dimensional output vector of MLP module to a single QoS value.

For all experiments, we use Huber loss [54] to train the model. The loss function is defined as

where $y$ is the ground truth QoS value, and $\hat{y}$ is the QoS value predicted by the model. We use threshold $\delta=1.0$. All models are trained using the Adam optimizer [55] with learning rate of $10^{-4}$ and batch size of 256.

We measure the Mean Average Error (MAE) and Root Mean Squared Error (RMSE) on the testing set to measure the performance of the model. They are defined as

and

where $(y_{i},\hat{y}_{i})$ is a pair of ground truth and predicted QoS values, and $N$ is the number of data points in the testing set. We measure the MAE of the model at the end of each training epoch, and use the one with lowest MAE as the final trained model. We observed that for throughput experiments, the final model can be obtained before 1,500 epochs. For response time experiments, the final model can be obtained before 600 epochs.

In our experiment, we selected 7 baselines to compare with our model and evaluate its performance. This includes traditional algorithms and the latest research. We keep dataset split method and evaluation settings for all baselines same to the proposed llmQoS for fair comparison.

UIPCC [18] is a traditional domain-based collaborative filtering method that predicts missing QoS values by leveraging historical invocation information of users and services.

RegionKNN [56] is a hybrid collaborative filtering algorithm that considers geographic information for service recommendations.

LACF [57] is a location-aware collaborative filtering method that integrates user and service location information to recommend web services to users.

PMF [58] is an enhanced traditional matrix factorization method that incorporates probabilistic factors into matrix decomposition for QoS prediction.

BGCL [31] is a graph-based contrastive learning method that utilizes graph attention mechanisms to incorporate domain information of users and services for QoS prediction.

LMF-PP [59] is a location-based matrix factorization method that integrates invocation and domain information, and utilizes fused similarity for preference propagation, addressing cold-start issues to achieve QoS prediction.

DCALF [60] detects domains and noise of users and services from the original QoS matrix, introducing density peak-based clustering methods during modeling for simultaneous detection of neighborhood and noise, achieving QoS prediction.

The performance of the proposed llmQoS is compared with the baseline methods in Table I. Phi3mini is used for llmQoS in this experiment. We compare both MAE and RMSE for throughput and response time at different densities. It is evident that llmQoS outperforms all baselines consistently at all densities. For throughput value prediction, llmQoS can reduce the MAE by more than 20%, and RMSE by more than 6% from the best baseline model. For response time value prediction, llmQoS can also reduce MAE for more than 10% from the best baseline method. The results demonstrate that with the rich features from the descriptive sentences extracted by LLMs, the QoS prediction model can effectively find the similarity and correlation among users and services. Along with the information learned from historical interactions, llmQoS can achieve high QoS prediction accuracy.

When setting the training epochs, we configured 1500 epochs for throughput prediction and 600 epochs for response time prediction. We plotted training curves to validate these settings. Figure IV illustrates the loss, MAE, and RMSE curves of the llmQoS model during training at sparse densities ranging from 5% to 20%. As the number of iterations increases, the model gradually converges to optimal performance. We observe that the model reaches optimal performance for throughput prediction before 1500 epochs, while for response time prediction, it reaches optimal performance before 600 epochs.

As discussed in § II-B and § III-A, larger LLMs have higher capacity and can potentially achieve higher performance on various tasks. So here we compare the performance of llmQoS with different LLMs as the feature extractor. We compare the larger and more recent Phi3mini with the smaller and older RoBERTa model. In addition, we remove all LLM feature related components from llmQoS, and only use the ID embedding vectors of users and services to train a QoS prediction model. This model is referred as “ID only” model. We train all models using the same training hyper-parameters, and the results are compared in Figure V.

It is clear that adding LLM features to QoS prediction is helpful, and can reduce the prediction error significantly from the ID only baseline. This further verifies that the information from the attributes of users and services is useful for service recommendation. And Phi3mini achieves lower MAE and RMSE than RoBERTa consistently on all densities. This shows that the superior capability of larger LLMs on NLP tasks also transfers to QoS prediction task.

However, it is important to note that larger LLMs runs at a higher computational cost. Phi3mini is considered one of the smallest modern LLMs, as the larger ones have tens or hundreds of billions of parameters. Larger LLMs can also have higher feature dimensionality, making training the QoS prediction network slower. In practice, trade-off needs to be made between prediction accuracy and computational cost.

In § IV-F we have shown that LLM feature improves QoS prediction accuracy, and larger Phi3mini is more effective than smaller RoBERTa. However, the prediction model with LLM feature input has more trainable parameters than the model that only uses ID embedding. What is more, since the feature vectors from Phi3mini have 3,072 dimensions compared to RoBERTa’s 768 dimensions, the projection layer for Phi3mini also has more parameters than RoBERTa. So it is plausible that the performance improvement is due to the increased number of parameters resulting in large model capacity.

To verify this, we train a model that still has 3,072-dimensional feature vectors as input. But those feature vectors are randomly initialized for each user and service, so they do not contain any useful information. However, this model has the same number of parameters as the llmQoS model with Phi3mini feature. We compare the QoS prediction performance of this model with both the ID only model as in § IV-F and the llmQoS with Phi3mini in Figure VI. Generally, the model with random feature does not outperform ID only model. On throughput dataset under $D$=5% and $D$=15%, the accuracy is marginally higher than the ID only model. But under other settings the performance actually worsens. We also observe that with random feature, the model converges with much less epochs than using Phi3mini feature. This indicates that more trainable parameters lead to overfitting. We verify that the performance boost of llmQoS is from the useful information encoded in the LLM feature, instead of added parameters of the model.

To if LLM features can benefit QoS prediction in general, we further conduct experiments by incorporating LLM features into another QoS prediction model. We selected the Bi-subgraph network based on graph contrastive learning (BGCL) model [31], which has a more complex network architecture. BGCL uses location attributes of users and services for graph contrastive learning, as well as attention mechanism to aggregate features. If LLM features can improve the QoS prediction accuracy for BGCL, we claim that the benefits of using LLM can be further demonstrated.

We keep all other features and network architecture unchanged for BGCL, with the only addition of the LLM feature vectors along side with the embedding vectors of BGCL at the very beginning of the network of BGCL. Similar to § IV-F, we compare BGCL with Phi3mini, BGCL with RoBERTa, and vanilla BGCL. The experimental results, as shown in Figure VII, are similar to the results in Figure V too. We further verify that that integrating LLM features significantly enhances the model’s performance, and larger model achieves more satisfactory results. This demonstrates that LLM features are beneficial to QoS prediction problem, regardless of the network architecture. And we expect this to apply to other QoS prediction models as well.

We test the performance of llmQoS using different configurations for the MLP model as described in § III-B. For this experiment we test the density level $D$=5%, and using Phi3mini as the LLM feature extractor. We vary the depth of the MLP by adding or removing layers from it. The QoS prediction performance of different configurations is shown in 8(a) and 8(b). We change the width of the MLP by multiplying a width factor to the dimensionalities of the layers in it. The QoS prediction performance of different configurations is shown in 8(c) and 8(d).

It can be observed that the depth and width of the MLP network have a minor impact on the QoS prediction performance. In most cases the change in MAE and RMSE is under 5%. However, when the MLP depth is set to 1, or the width factor is set to 0.5, there is a significant deterioration in the performance. This is caused by the network is too shallow and narrow, resulting to very low capacity. Considering this paper primarily focuses on using large language models to assist QoS prediction, we have selected a reasonably appropriate depth and width for the MLP network without extensive searching.

We test the performance of llmQoS trained using different batch sizes and learning rates. For this experiment we test the density level $D$=5%, and using Phi3mini as the LLM feature extractor. The same configuration of MLP used in the main experiments is used.

The QoS performance of models trained using different batch sizes is compared in 9(a) and 9(b). Generally speaking, the impact of batch size is inconsistent and insignificant. Only for throughput, larger batch size of 1024 causes the prediction accuracy to drop significantly. The performance with different learning rates is compared in 9(c) and 9(d). Learning rate has a huge impact on the model’s prediction accuracy. Very small learning rate results in slow convergence, while large learning rate causes unstable training. Both causes low performance. The optimal learning rate is around $10^{-4}$ as used in the main experiments.