FACOS: Finding API Relevant Contents on Stack Overflow with Semantic and Syntactic Analysis

Two steps are involved in finding the API mentioned in informal text content: (1) API mentions extraction, and (2) API mentions disambiguation. API mention extraction aims to identify common words that refer to the APIs. API mentions disambiguation, on the other hand, links API mentions with the APIs they reference. DATYS [6] specifically deals with the API mention disambiguation via type scoping in the informal text of Stack Overflow to resolve ambiguous mentions of Java API methods, where the mentions have been identified.

After extracting API method candidates from input Java libraries, DATYS scores API method candidates based on how often their types (i.e., classes or interfaces) appear in different parts (i.e., scopes) of the Stack Overflow thread with identified API mentions. Having a type that appears in more scopes will increase the API candidate score. Here, DATYS considers three scopes: Mention scope, which covers the mention itself. Text scope, which covers the textual content of the thread, including the mentions. Code scope, which covers the code snippets in the thread. API candidates are ranked according to their scores for each API mention in the thread. DATYS takes the top API candidate with a non-zero score as the mentioned API. If the leading API candidate has a zero score, DATYS considers the mention as an unknown API. Luong et al. built a ground truth dataset containing 807 Java API mentions from 380 threads in Stack Overflow.

CodeBERT [9] was developed using a multilayered attention-based Transformer model, BERT [10]. As a result of its effectiveness in learning contextual representation from massive unlabeled text with self-supervised objectives, the BERT model has been adopted widely to develop large pre-trained models. Thanks to the multilayer Transformer [11], CodeBERT developers adopted two different approaches than BERT to learn semantic connections between Natural Language (NL) - Programming Language (PL) more effectively.

Firstly, The CodeBERT developers make use of both bimodal instances of NL-PL pairs (i.e., code snippets and function-level comments or documentations) and a large amount of available unimodal codes. In addition, the developers have pre-trained CodeBERT using a hybrid objective function, which includes masked language modeling [10] and replaced token detection [12]. The incorporation of unimodal codes helps the replaced token detection task, which in turn produces better natural language by detecting plausible alternatives sampled from generators.

Developers trained CodeBERT from Github code repositories in 6 programming languages, where only one pre-trained model is learned for all six programming languages with no explicit indicators used to mark an instance to the one out of six input programming languages. CodeBERT was evaluated on two downstream tasks: natural language code search and code documentation generation. The study found that fine-tuning the parameters of CodeBERT obtained state-of-the-art results on both tasks.

Our goal is to find Stack Overflow threads that mention a given API method¹¹1In this paper, we use the terms API, method, and API method interchangeably.. Specifically, given an API method, we strive to find Stack Overflow threads containing words matching the simple name of the given API method. In Java, the simple name of an API method is the name of the method without the class and the package names. For example, m is the simple name of API method com.example.Class.m. We want to classify whether the threads having the simple name m is actually relevant to API method com.example.Class.m. In summary, the task is defined as: “For each API method in a set of given API methods, identify Stack Overflow threads that refer to it.”

The pipeline of FACOS is presented in Figure 1. It is divided into 2 main steps:

(1) Collecting various API-related resources from a given API method name; and (2) Recommending relevant threads using the collected API-related resources.

In the step (1), FACOS finds Potential Threads from Stack Overflow using the simple name of the given API method as the query. Potential Threads are the threads that have at least one word matching with the simple name of given API. The API method comment and implementation code are directly obtained from the source code repository of the given API. Last but not least, the API Candidates are obtained from a database of API methods. The API Candidates are API methods that have the same simple name as the given API method.

The objective of step (2) is to identify whether each Stack Overflow thread in the Potential Threads actually refers to the given API. FACOS has two components: API relevance classifier and DATYS+. API relevance classifier is designed to draw the relevance between a thread and an API method by capturing the semantic similarity between (1) paragraphs and code snippets in the thread; and (2) API method comment and implementation code. API relevance classifier outputs a semantic relevance score representing the relevance it measures. In contrast, DATYS+ outputs a syntactic relevance score based on the existence of the terms from the fully qualified name of the given API in different scopes of a Stack Overflow thread. For example, API ”A.B.C” has terms such as ”A”, ”B”, and ”C”. The last term, ”C”, is the simple name of the API method. The second last term, ”B”, is the type of the API method. Both DATYS and DATYS+ use type scoping [6] to give score based on the existence of the type of the API (i.e., ”B” in the example) in different scopes of the Stack Overflow thread (code scope, text scope, etc.). However, type scoping of DATYS+ is modified to be suitable for the search task and we are going to describe it in Section IV-A. It outputs a score that indicates the syntactic relevance between the given API and the thread, we call it DATYS+ score.

After step (2), each thread will have a score indicating if the thread refers to the given API method. This score is combined from semantic relevance score and DATYS+ score and is called joint relevance score. Threads predicted as referred to the given API method are then returned to the user. We describe FACOS components (i.e., DATYS+ and API relevance classifier) in detail in Section IV.

DATYS+ is an extension of DATYS. DATYS used regular expressions to capture the types of API method invocations available in code snippets of the thread. However, these regular expressions are limited and thus DATYS may miss some mentions in code snippets. To capture more types, DATYS+ modifies the type scoping algorithm by adding a new score.

Algorithm 1 indicates how modified type scoping works. Compared to DATYS’s, DATYS+’s type scoping algorithm receives $\mathit{CodeSnippets}$ as another input. $\mathit{CodeSnippets}$ represents the content available in code snippets of the Stack Overflow thread. In addition, inputs of the original type scoping algorithm are also considered. $\mathit{APIMention}$, $\mathit{PTypeList}$, $\mathit{APIMethodCandidate}$, and $\mathit{ThreadContent}$ stand for the simple name of the given API, the list of possible types extracted from code snippets following the algorithm used by DATYS, the API Candidates, and the thread’s textual content (i.e., title, text, tags), respectively. The three scopes used by DATYS are also used in DATYS+. In Mention Scope (Lines 3-8), DATYS+ increases an API score if its type appear within the API mention. In Text Scope (Lines 10-13), DATYS+ increases an API score if its type appear within the textual content of the thread. In Code Scope (Lines 17-21), DATYS+ increases an API score if its type matches with the type of method invocation or imported types in the code snippet. Additionally, in Code Scope, DATYS+ also looks at the content of the code snippets and increases the API score of the corresponding API candidate if there are tokens in the code snippets that match with the API type (Lines 14-16). This score helps to capture the occurrence of types that would be missed by a more accurate matching used in DATYS. Thus, we call the scope of this score Extended Code Scope

After executing type scoping, DATYS+ returns scores for the API Candidates. The scores are then normalized to a range of [0, 1] following the minimum and the maximum score from the API Candidates. DATYS+ then takes the normalized score of the given API method and passes it to the next step.

We follow the process described in Figure 2 to build API relevance embedding. Firstly, each thread in Potential Threads needs to be converted into an embedding. A thread may contain m paragraphs and n code snippets. A paragraph is a piece of textual content on a Stack Overflow thread that is separated from other contents in the thread via a newline character. Code snippet is a piece of code content on a Stack Overflow thread. It is typically enclosed with a starting tag $\langle pre\rangle\langle code\rangle$ and an ending tag $\langle/code\rangle\langle/pre\rangle$. Each paragraph is paired with each code snippet to create a pair of thread content. Therefore, a Stack Overflow thread would have $m\times n$ thread content pairs. A natural-programming language model, CodeBERT²²2https://github.com/microsoft/CodeBERT, is used to extract the semantic meaning of each thread content pair. It encodes the $m\times n$ thread content pairs into $m\times n$ thread embeddings. thread embedding is the representation vector of thread content that created by CodeBERT’s encoder. By converting the pairs from a textual form to a numerical vector form with a pre-trained CodeBERT model, the semantic relationship between the paragraphs and code snippets is extracted. Before feeding the thread content pairs into the encoder of CodeBERT, each pair is pre-processed following the format:

Secondly, to build API relevance embedding, API comment and implementation code also need to be converted into an embedding. The API method comment is a piece of textual content that describes the functionality of the API method and how to use it. The API implementation code is the code inside the API method body that implements the described functionality. The API comment and implementation code are extracted from the Javadoc and the JAR files, respectively, they are pre-processed to the following format:

They are then transformed into a numerical representation vector via the CodeBERT encoder.

Finally, each thread embedding vector and the method embedding vector are then concatenated to a vector. We call this concatenated vector API relevance embedding. In total, $m\times n$ API relevance embedding vectors would be created.

The API relevance classifier is a binary classifier that utilizes a neural network with two fully connected layers to predict whether the API relevance embedding comes from a Stack Overflow thread that refers to the given API method.

The API relevance classifier has two modes of operation: training and deployment modes. In the training mode, the API relevance embeddings are used to train the API relevance classifier. When there is an imbalance between positive and negative labels, API relevance classifier upsamples the minority label. Whenever the thread refers to the given API method, all API relevance embedding created from the thread would be considered as positive by the classifier. Otherwise, in case the given API method is not referred to by the thread, every API relevance embedding of the thread would have negative labels. In the deployment mode, API relevance classifier produces probability scores for the $m\times n$ API relevance embedding. These scores are averaged and passed to the next step. The averaged score indicates the likelihood that the thread refers to the given API.

We follow the process in Figure 3 to compute the joint relevance score. DATYS+ and API relevance classifier output scores $A{}$ and $B{}$, respectively. Both represent their confidence that the given API method is mentioned in the thread. The two scores are then combined to a joint relevance score $C$ following this formula:

The value of $x{}$ will be estimated based on the training data. In detail, we let $x{}$ increase gradually from $0$ to $1$ with a step of 0.1. There are ten possible values of $x{}$: $\{0,0.1,0.2,...,0.9,1.0\}$. The value of $x{}$ giving the highest performance in the training data is then chosen.

We utilize the dataset provided in DATYS work [6] to evaluate both FACOS and DATYS. We split 380 Stack Overflow threads to 253 training threads and 127 testing threads with the ratio of 2:1. The training threads are utilized to train the API relevance classifier while the testing threads are used to evaluate FACOS and DATYS. Next, as mentioned in Section IV-B, for each Stack Overflow thread in the training threads, we extract its thread embeddings and these thread embeddings are grouped into a training set. Similarly, for each Stack Overflow thread in the testing threads, we extract its thread embeddings and these thread embeddings are grouped into a testing set. The numbers of API relevance embeddings of the dataset are shown in Table I. The numbers of the embeddings in training set and testing set are $57\,690$ and $26\,212$, respectively.

To generate API relevance embeddings for the API relevance classifier for training, for each thread, if the given API appears in the thread, we generate API relevance embeddings for thread contents and method contents as described in Section IV-B. These embeddings would have positive label because they are created from the API that is referred to by the thread. To generate embeddings with a negative label for a thread, we find APIs that have the same simple name as the given API and are not mentioned in the thread. We then create API relevance embeddings from these APIs and label these API relevance embeddings as negative.

To train the API relevance classifier, there are $344$ APIs. These APIs are used to generate the training method embeddings. In the testing set, there are $181$ APIs. These APIs are used to generate the testing method embeddings. Table II shows the numbers of positive and negative API relevance embeddings created in training and testing sets. The number of negative API relevance embeddings is approximately 4 times more compared to the positive ones in the same set of thread. Due to this imbalance, positive API relevance embeddings are randomly up-sampled to balance the two classes within the API relevance classifier training process.

The API relevance classifier is trained using 6 epochs on the training data. After the first 6 epochs, the value of the loss function has relatively converged. The learning rate of the training is set to $10^{-3}$.

To evaluate the proposed approach on identifying threads that are relevant to an API, we use three metrics: Precision, Recall, and $F_{\textit{1}}\textit{-}score$. In order to calculate the three aforementioned metrics, True Positive, False Positive, and False Negative should be defined first. Our task focuses on finding threads that actually refer to a given API. True Positive is the case where a thread is deemed to be relevant by the approach is indeed relevant. False Positive is the case where the thread that is deemed to be relevant by the approach is actually irrelevant. False Negative is the case where a threads is deemed to be irrelevant by the approach is actually relevant. The metrics are calculated using the following formulas:

Table III shows the performance of the DATYS and FACOS in finding threads that are relevant to the given API. FACOS, in general, outperforms DATYS. On average, FACOS achieves an F1-score of 0.873, which is an improvement of 13.9% compared to DATYS. FACOS also beats DATYS in terms of precision and recall.

Table IV shows how well each component in FACOS is. Since the “API relevance classifier-only” version of FACOS gives worst result, API relevance classifier may not be able to resolve the task well independently. Partly, this might be be due to the limited amount of the training data (i.e., only 253 training threads). In addition, the “DATYS+ only” version of FACOS performs much better compared to the “API relevance classifier-only” version. However, FACOS is still better than both of them. It demonstrates that both components are useful and essential.

Table V shows the performance of FACOS in the training set when we vary the values of weighting factor. The bold numbers in each row of the table are the average $F_{\textit{1}}\textit{-}scores$ of the chosen values of x in the training sets. Similarly, Table VI the performance of FACOS in the test set when we vary the values of weighting factor. The highest $F_{\textit{1}}\textit{-}score$ for both the training and testing set is achieved when the value of the weighting factor is equal to 0.3. It demonstrates that our strategy to pick the value of the weighting factor that leads to the best performance in the training data works really well.

From the content of the thread, it would be difficult to find the relevance between the text written in the paragraphs and the given API method (i.e., org.mockito.stubbing.OngoingStubbing.thenReturn) since the type (i.e., OngoingStubbing) does not appear in the thread. The text only shows the user view towards the code snippet without having a description mentioning the application or usage of the observed API method invocation (e.g., thenReturn in the code snippet of Figure 4). Sentences such as ”This works like charm!” do not provide much information to identify whether the observed API method refers to the given API.

Therefore, we leverage the content of the thread which might be relevant to the content of the API method. For example, in the thread above, its title which is shown in Figure 6(a), ”Optional cannot be returned by stream() in Mockito Test classes”, relates to the comment of the given API which is Sets a return value to be returned when the method is called in Figure 6(b). Due to this feature, FACOS can successfully consider this thread as relevant while DATYS missed it.

The API method and relevant information search does not always utilize syntactic information. For some libraries, some methods are chained (as in Figure 4), so the type is not displayed. It requires a type system to determine its type. In addition, the Type system is not 100 percent reliable due to a lack of import information, variables, etc. It is ineffective to use the similarity of word representations provided by language models for API contents search. API contents search is unique to other tasks such as code search. In code search, due to the term ”transfer” and ”convert” being related to currency, the method named ”transferSGDToUSD” might be the answer to the query ”convert SGD to USD.” However, searching for an API method requires explicit mention of the API name in the query, which could be difficult as different APIs may share the same API method names. To solve the problem, solely using a model that only learns the syntactic meaning of textual and code content is insufficient. Therefore, we incorporated semantic information for API content search, which generated better results, as we have demonstrated from the results of our experiment.

A threat to internal validity is related to experiment bias. We obtain our dataset from another work. We also run the baseline using the code provided by the author. We then check our code multiple times to ensure that we do not make mistakes. We believe there should be little threats to internal validity. We also release the dataset and the code for our experiments for all to use.

A threat to external validity is on whether the approach is applicable to other platform other than Stack Overflow. This experiment mainly focuses on Stack Overflow and Java programming language, therefore, it is uncertain whether FACOS can be applied on other discussion venues that also talk about API issues. The potential platform might be Reddit, which has sub-reddit (i.e., a place gathering Reddit’s threads discussing a particular problem) discussing programming languages and frameworks. It has title, textual and code content as same as Stack Overflow. Thus, the similarity suggests that we can potentially apply FACOS to Reddit too. We leave this possibility for a future work. Regarding the threat that changing targeting programming language would affect the accuracy of FACOS, although we focus only on Java, the features (e.g., API comment/documentation, API implementation code, fully qualified name, class/type name, etc.) required for the approach can be found from other programming languages. Therefore, we leave this as a future work to study whether it may work well when applied to other programming languages.

There is a also threat of construct validity on whether precision, recall, and $F_{\textit{1}}\textit{-}score$ is a suitable evaluation metric for our task. Our task is a classification task. Many work in software engineering has used precision, recall, and F1-score as the evaluation metric for classification task [6, 14, 15, 16]. Thus, we believe the threats are minimal.

Many past works [14, 17, 18, 19, 20, 21, 22, 23] deals with API disambiguation. There are two main groups: informal text disambiguation [18, 19, 20, 21, 14] and code snippet disambiguation [22, 23, 17]. As suggested by its name, the first aims to disambiguate API mentions in textual content while the second deals with disambiguating API mention in code snippets.

For informal text disambiguation, several work utilizes classical information retrieval approaches such as Vector Space Model and Latent Semantic Indexing to disambiguate the API mentions [18, 19, 20] while some others use heuristics [21]. Bacchelli et al [20] combined string matching and information retrieval algorithms to link emails to source code entities. Dagenais and Robillard [21] identified Java APIs mentioned in support channels (e.g., mailing list, forums), documents, and code snippets. Ye et al. [14] worked on API disambiguation in the textual content of Stack Overflow thread by utilizing mention-mention similarity, mention-entry similarity, and scope filter. Luong et al.[6] used type scoping to disambiguate the API mentions in Stack Overflow thread.

The work on API disambiguation on Stack Overflow thread can be viewed as another side of the coin of the task in finding threads that are relevant to the API. When we disambiguate an API in a thread, the disambiguated API is relevant to the thread as the thread is talking about the API.

Several studies have explored how to search for the code for API and related information retrieval. Lv et al. [24] proposed Codehow to deal with the lack of query understanding ability of the existing tool. By expanding a user query with APIs, Codehow can identify potential APIs and perform a code search based on the Extended Boolean model, which considers the impact of APIs on code search. Gu et al. [25] proposed DeepAPI to search for API usage sequences. As opposed to assuming a bag of words, it learns the sequence of words within a query and the sequence of APIs associated with it. DeepAPI encodes a single user query into a fixed-length context vector to generate an API sequence.

Other studies have also exploited different aspects of APIs and natural language to better retrieve the APIs and their related information. The techniques include: using global and local contexts of the queries [26], leveraging usage similarity for effective retrieval of API examples [27], employing word embeddings to document similarities for improved API retrieval [28], exploiting user knowledge [29], and task-API knowledge gap [30] during retrieval of semantically annotated API operations.

Wang et al. [31] developed a transformer-based framework for unifying code summarization and code search. Shahbazi et al. [32] proposed API2Com to improve automatically generated code comments by fetching API documentations. Alhamzeh et al. [33] built DistilBERT-based argumentation retrieval for answering comparative questions. Dibia et al. [34] and Vale et al.[35] developed a usable library for question answering with contextual query expansion and a question-answering assistant for software development using a transformer-based language model, respectively. Ciniselli et al.[36] performed an empirical study on the usage of Transformer Models for code search and completion.

Our study also work on API resource retrieval. Specifically, we retrieve Stack Overflow threads that are relevant to a target API that we are searching for.

Treude et al. [37] studied the augmenting API documentation with insights from stack overflow. [4] explored the Crowd documentation by examining the dynamics of API discussions on Stack Overflow, whereas [38] dubbed the Stack Overflow as the Social Media for Developer Support in terms of provided utilities. [39] and [40] worked on classifying stack overflow posts on API issues and contextual documentation referencing on stack overflow. The dichotomy of these studies is notable where some research like [41] and [42] studies how API documentation fails via the API misuse on stack overflow, other studies [43, 44] heavily lean on the Crowdsourced knowledge on stack overflow for automated API documentation with tutorials. Similarly, crowdsourced knowledge was hailed by [45] and [46], who explored the innovation diffusion and web resource recommendation for hyperlinks through link sharing on stack overflow.

Our work support the effort in this line of study. FACOS can automatically find threads about a particular API in Stack Overflow that can be augmented to the corresponding API documentation.

There are several works focused on disambiguation task [21, 22, 47, 23]. We also have found a variety of word sense and entity disambiguation methods employed for different objectives [48, 49, 50, 51, 52, 53, 54, 55, 56]. These studies have solved myriads of problems via solving lexical disambiguations in literature. The task of word sense disambiguation is to identify a target word’s intended meaning by examining its context. Researchers have used Word Sense Disambiguation to predict election results by enhanced sentiment analysis on Twitter data. Researchers have associated place-name mentions in unstructured text with their actual references in geographic space using word disambiguation. Other research has also proposed unsupervised, knowledge-Free, and interpretable Word Sense Disambiguation for various applications. Researchers used this approach to add meaning to social network posts when it comes to named entity recognition and disambiguation. Different Entity-fishing tools were also developed for facilitating the recognition and disambiguation service. In recent years, tools that allow researchers to recognize and extract named entities have become increasingly popular.