DySR: A Dynamic Representation Learning and Aligning based Model for Service Bundle Recommendation

The main idea of NLP-based approaches is to make recommendations by measuring the similarity between the description of the candidate services and the requirement. Keywords[14, 15] and TF-IDF[16] are used to match services to satisfy mashup creation. However, these matching procedures cannot really understand semantic and suffer from poor performance.

Ontology-based approaches annotate requirements and services with domain ontologies and use these ontologies to match high-level concepts or calculate their semantic similarities[17, 18, 19]. However, the lack of suitable domain ontologies and the huge manual annotation cost of ontology construction make ontology-based approaches difficult to use in real-world scenarios. Some studies attempt to use latent semantics to denote text features. Currently, the most common technique used to solve this problem in the field of service computing is the topic model[4, 10]. For example, Li et al. [20] use a topic model to explore the semantic relationships between mashups and services. Zhong et al. [21] refactor descriptions by using Author-Topic Model [22] to eliminate the gaps between mashups and services. However, small data volumes, high noise in data, and ignoring word orders make the performance of topic model-based methods unsatisfactory; for example, the recent SOTA method [10] based on topic models has an $F1@5$ value of only about 30% on the ProgrammableWeb dataset. With the great success of pre-trained language models (PLMs) and deep learning models in the NLP, some researchers start using PLMs and deep learning to solve the service recommendation problem. For example, Bai et al. [23] designed a stacked denoising autoencoder (SADE) to extract features for the recommendation. However, experimental results show that the simple introduction of PLMs and deep learning does not significantly improve recommendation performance, possibly due to the inability of existing models to directly adapt to service recommendation and negative transfer [24].

The major drawback of the NLP-based approaches is the inability to leverage the social information between services, leading to recommended service bundles with combinations of services that cannot co-occur due to real-world constraints, despite being functionally compatible.

Unlike NLP-based approaches, graph-based approaches make recommendations by mining information from historical interaction between services or requirements (users). For example, [25, 5, 26] design recommendation algorithms based on the feature learned on knowledge graph.

The graph-based approach usually works in conjunction with collaborative filtering (CF). For example, Chen et al. [27, 28] propose a neighborhood integrated matrix factorization approach to predict the quality of service (QoS) of candidate services. Chang et al. [6] designed a graph-based matrix factorization approach to predict QoS, then use the QoS to select services. Besides predicting QoS, graph is also applied to find similar users or services. For example, Maardji et al. [29] proposes a frequent pair mining method for mashup development. Qi et al. [30] adopts a hybrid random walk to compute the similarities between users or services, and a CF model is designed for service recommendation. [31, 32] build a heterogeneous information network using various information of services and mashups to measure the similarity between mashups, and then use the user-based CF to ranking candidate services.

Graph-based approaches cannot solve the cold start problem, as new requirements do not have any historical information.

Considering the complementarity of textual and graph information, a number of hybrid service recommendation approaches combining the two types of data have been proposed in recent years.

Li et al. [20] add the invocation relations between requirements and services to a latent Dirichlet allocation (LDA) model to enable the topic model learn the relationship between services and requirements, . [8, 16] incorporates data structure made up of service and their co-invocation records into LDA. Jain et al. [33] and Samanta et al. [34] use topic models and neighbor interaction probabilities to calculate similarity scores between services and requirements, then multiply these scores to rank candidate services.

Deep learning-based approaches are becoming the mainstream of hybrid methods. For example, Xiong et al. [35] integrates the invocation relations between services and requirements as well as their description similarity into a deep neural network (DNN). Chen et al. [36] propose a preference-based neural collaborative filtering[37] recommendation model, which uses multi-layer perceptron to capture the non-linear user-item relationships and obtain abstract data representation from sparse vectors. Ma et al. [9] utilize the powerful representation learning abilities provided by deep learning to extract textual features and features from various types of interactions between mashups and services. Wu et al. [38] propose a neural framework based on multi-model fusion and multi-task learning, whiche exploits a semantic component to generate representations of requirements and introduces a feature interaction component to model the feature interaction between mashups and services.

Although the hybrid approach is a significant improvement over the NLP-based approaches and graph-based approaches, the current SOTA approach[9] still has an F1@5 value below 40% on the ProgrammableWeb dataset. This is mainly caused by ignoring the difference in representation between services and requirements and the evolving service social environment. The DySR model proposed in this paper is a hybrid approach that achieves $F1@5$ values close to 70% on the ProgrammableWeb dataset by tackling the representation gap and service evolution.

As shown in Fig. 2, the proposed DySR model consists of two subtasks: a supervised service bundle recommendation task and an unsupervised evolving service representation task. The service recommendation subtask is used to satisfy our need to recommend suitable services for a new requirement, and this task is used in both the training and inference stage of the DySR model. The representation gaps between requirement and service are eliminated, and the requirement-service matching function is learned in this subtask. The evolving service representation subtask is used to solve the service evolving problem. It is a supplementary task to the service recommendation subtask and is used to provide a suitable service representation for service bundle recommendation, so it will only be used in the training stage of the DySR model.

During the training stage, the input of DySR model is a requirement $r$ that is satisfied at historical time $t$ by a set of component services $C^{+}=\{s_{1},s_{2},\dots,s_{n}\}$, which is used in the supervised task as ground truth. And a set of co-invocation event $\mathcal{O}=\{o=(s_{u},s_{v},t)|s_{u},s_{v}\in C^{+}\text{ and }s_{u}\neq s_{v}\}$ is generated from $C^{+}$ as the training samples of the unsupervised task. While in the inference stage, the input of DySR model is a requirement $r$, and the output is a set of recommended component services $\hat{C}=\{\hat{s}_{1},\hat{s}_{2},\dots,\hat{s}_{m}\}$.

For a requirement $r$ at time $t$, we denote the latent representation as $\mathbf{v}_{r}\in\mathbb{R}^{d}_{r}$, a dense vector obtained by a certain encoding method (e.g. text encoding using pre-trained language model), where $d_{r}$ is the dimension of the requirement’s latent representation. We use $\mathbf{z}^{\bar{t}}_{s}\in\mathbb{R}^{d_{s}}$ to denote the recently updated latent representation of candidate service $s$ just before $t$, where $d_{s}$ is the dimension of each service’s latent representation. And $Z^{\bar{t}}$ is all candidate services latent representation. The obtaining of $Z^{\bar{t}}$ is discussed in detail in Section III-C.

Unlike existing approaches that maintain a requirement-service invocation graph, DySR only needs to maintain a dynamic service co-invocation graph, which is more practical:

• •

  Compared to services, requirements are one-off and grows rapidly in volume, making the requirement-service invocation graph very large and sparse over time. On the one hand, this does not facilitate the extraction of information, and on the other hand, requires more expensive hardware resources.

• •

  Requirements are usually created by users, which may involve privacy and security issues. For example, users may not want their requirements to be available to other users. Service co-invocation graph can solve this problem, as it only records co-invocations between services but not specific requirement content.

The service co-invocation graph can be represented as a dynamic adjacency matrix $A^{t}\in\mathbb{R}^{|\mathcal{C}^{t}|\times|\mathcal{C}^{t}|}$, where $\mathcal{C}^{t}$ is the set of services at time $t$, and $A^{t}_{s_{u},s_{v}}$ is the number of times service $s_{u}$ and $s_{v}$ have been co-invoked before time $t$. The phenomenon of service evolution can be reflected by the evolution of the service co-invocation graph, which can be seen as a co-invocation event-driven temporal point process. We use $o=(s_{u},s_{v},t)$ to represent a co-invocation event, which means $s_{u}$ and $s_{v}$ are co-invoked at time $t$.

We use a variant of the DyRep[40] model to learn the service representation on a dynamic service co-invocation graph. For a given co-invocation event $o=(s_{u},s_{v},t)\in\mathcal{O}$, the main training steps consist of the following:

To model the process that requirements trigger service evolution and that service evolution reacting the selection of services by requirements, we adopt a GAN-style training step. Specifically, we use two independent Adam optimizers[41] $opt_{1}$ and $opt_{2}$ to optimize the parameters $\{\Psi,\Lambda\}$ in the supervised service recommendation task and parameters $\Omega$ in the unsupervised evolving service representation task. It is also important to note that we perform supervised task optimization for each batch of samples before unsupervised task optimization. . The pseudo-code for the GAN-style training algorithm is given in Algorithm 2.

We evaluate the proposed DySR model on the real-world ProgrammableWeb dataset, which is also the dataset used in existing service recommendation studies.

ProgrammableWeb: The dataset is the largest online Web service registry. We collected a total of 23,520 APIs and 7,947 mashups on Oct 10, 2020. The mashups without functional description, the services that have not been invoked, and the mashups with fewer than two component services were removed. The experimental dataset contains 3,380 mashups, whose functional descriptions are used as requirements, and 720 APIs. We sorted the mashups by when they were created, and we initialize the adjacency matrix $A$ using the co-invocations from the earliest $300$ mashups. The next $2,400$ mashups are used as the training set, and the remaining $680$ mashups are used as the test set.

We adopted the following evaluation metrics to measure the recommendation performance:

where $R$ is the set of requirements in the test set and $|R|$ denotes the size of $R$. For requirement $r$, $\hat{C_{r}}$ is the recommended services, while $C_{r}$ is its actual component services.

We use bert-base-uncased provided by Transformers[42] to obtain requirement representation with the dimension $d_{r}$ set to $768$, and it should be noted that we do not do fine-tune on bert-base-uncased. Service representations $Z^{0}$ are initialized by a $128d$ Word2Vec[43] word embedding trained on text8⁴⁴4http://mattmahoney.net/dc/text8.zip. We use the two different pre-trained language models to reflect the representation gap between requirement and service. It should be noted that the initial representation of the service in the DySR model can theoretically be obtained using other information or in a random way. We implement a DySR variant called DySR-Rand, which does not need any prior knowledge of service and uses a randomly initialized representation of the service.

Gradient clipping is used in $opt_{2}$ to avoid gradient explosion, and the clipping value is set to $100$. We do not use dropout, and batch size is set to $50$. We conduct five independent experiments for each approach to preventing serendipity, and early-stop is applied to avoid over-fitting. All the results reported are average results.

To evaluate the effectiveness of model, we select five state-of-the-art service recommendation approaches:

1. 1.

   AFUP[33]: This approach first leverage probabilistic topic models to compute relevance between a service and a given requirement. description. And then use collaborative filtering to estimate the probability of a service being used by existing similar requirements. Finally, the multiplies these two term based on Baye’s theorem to rank candidates service.

2. 2.

   SFTN[34]: This approach extend AFUP by using hierarchical dirichlet process (HDP) and probabilistic matrix factorization (PMF) to tackle cold start issues and usage history.

3. 3.

   PNCF[36]: This approach use multi-layer perceptron to capture the non-linear user-item relationships and obtain abstract data representation from sparse vectors. However, text features were not considered in their original version, so we constructed two variants: PNCF-HDP using HDP adopted in SFTN to obtain text features, and PNCF-Deep using a pre-trained language model to obtain text features.

4. 4.

   MISR[9]: This approach propose a deep neural network that can captures multiplex interactions between services and requirements to extract hidden structures and features for better recommendation performance.

It should be noted that most baselines do not provide official code, and we can only reproduce it as described in their papers. In some cases, we were not able to reproduce the results they reported, possibly due to different ways of dividing the dataset and missing important parameter values. In these cases, we chose to directly compare the results reported in [9].

In this section, we give an overview performance comparison between our proposed DySR and baseline approaches. Fig. 4 shows the performance comparison of different approaches, showing that the DySR and DySR-Rand outperform all the five baselines across all evaluation metrics.

The AFUP and PNCF-HDP performed the worst of all the baselines for the following two main reasons: 1) They use a topic model to extract service/requirement representation from the description text, which ignores the order of words and further leads to lost semantic information; 2) Rough handling of service historical usage information. The introduction of probabilistic matrix factorization allows SFTN to handle historical usage information somewhat better, but the poor service/requirement representations obtained by the topic model remain limiting to its performance. The PNCF-Deep and MISR perform better than the other baselines because they use a pre-trained language model to obtain better representations of the services/requirements. In addition, MISR performs the best of all the benchmark methods because it takes into account multiple types of interactions between services and requirements.

Although DySR-Rand does not require any a priori knowledge of the service. DySR-Rand obtains competitive results with DySR, which shows that our proposed model is well suited to eliminate the gap between requirements and services.

DySR and DySR-Rand require less information than the baselines, while their performance is significantly better than all baselines. For example, all baselines need to maintain a requirement-service invocation matrix, while DySR only needs to maintain a small-scale service co-invocation matrix, and MISR additionally requires the tags of the service and historical requirements, which are not needed in DySR. Compared to the best performing baseline approach MISR, which requires the most information, DySR (DySR-Rand) improves the $Precision@5$, $Recall@5$ and $F1@5$ metrics by 28.4% (25.8%), 34.1% (30%) and 33.2% (30%), respectively. The performance improvements mainly benefit from evolving service representation and transformation function that allows the use of time information and the acquisition of better representation of services and requirements in same vector space.

DySR unifies several components that contribute to its effectiveness in service recommendation. In this section, we provide insights on evolving service representation component and transformation component and how they are indispensable to the service recommendation by performing an ablation study on various design choices of DySR. We designed the following variants of DySR for comparison:

• •

  DySR-Static: In this variant, we do not perform unsupervised evolving service representation tasks so that the service representation remains initialised, i.e. $Z^{t}\equiv Z^{0}$.

• •

  DySR-Space: In this variant, we turn off transformation function (Eq. 1) and directly use the original requirement representation and the dynamic service representation as inputs to the requirement-service matching function (Eq. 2). The dimension of the corresponding requirement-service matching matrix $W_{\Lambda}$ sets to $d_{r}\times d_{s}$.

• •

  DySR-None: In this variant, we do not perform unsupervised evolving service representation task and turn off transformation function.

The comparison among the variants of our approach is shown in Fig. 5. Both DySR-Space and DySR-Static outperform DySR-None on $Precision@N$, $Recall@N$ and $F1@N$, suggesting that evolving service representation and transformation function do contribute to the recommended performance. DySR outperforms DySR-Space and DySR-static indicating that evolving service representation and transformation function are complementary.

We conduct a series of qualitative analyses to understand how evolving service representation and transformation function contribute to recommendation performances. We first compare our evolving service representation against the static content-based service representation. Fig. 6 shows the UMAP[44] embeddings leaned by DySR (right) and initialised by service function description. The visualization demonstrates the evolving service representation have more discriminative power as it can effective capture the distinctive and evolving patterns of service combinations (line clusters in Fig. 6(b)) as well as outdated sets of services (block clusters in Fig. 6(b)) with empirical evidence.

Fig. 7 explains how DySR eliminates the representation gap between service and requirement. Fig. 7(a) demonstrates the gap between the representation of requirements and the representation of service that we stated in Section I. Compared with Fig. 7(a), the requirement representation space is much closer to the space of service representation in Fig 7(b), which shows that the transformation function in DySR does eliminate the difference between requirement and service representations to some extent. The same conclusion can be obtained by comparing Fig. 7(c) and Fig. 7(d). Fig. 7(c) and Fig. 7(d) can also illustrate that evolving service representations reduce the probability of selecting outdated services by moving them away from requirement representation space thereby improving recommendation performance.