NBSearch: Semantic Search and Visual Exploration of Computational Notebooks

Computational notebooks (e.g., Jupyter Notebooks and RStudio) have recently emerged as a form of programming environments that support dynamic development. A notebook is broken down into cells that appear in two primary forms, including code cells that contain executable scripts and markdown cells that contain formatted text (supplementing the comments in code cells). While a notebook is presented linearly, the order of executing the cells can be arbitrary, and each cell can be executed multiple times as well as edited between different runs, exhibiting a highly nonlinear programming pattern.

The dynamic nature of notebooks provides much freedom for developers, allowing them to experiment with different techniques, try out alternative methods, write temporary code, and conduct exploratory data analysis (Kery et al., 2018). However, these features, at the same time, create challenges in code management, comprehension, and development. For example, messes in code may accrue during the process, and developers may lose track of their thought processes. To help track code history, several tools have been proposed, such as Variolite (Kery et al., 2017) and Verdant (Kery et al., 2019), which support fast versioning and visual exploration of history. Code Gathering Tools (Head et al., 2019) help find, clean, recover, and compare versions of code in Jupyter Notebooks. Further, BURRITO (Guo and Seltzer, 2012) and TRACTUS (Subramanian et al., 2020) provide the provenance of analysis workflows by capturing and displaying code outputs, development activities, users’ hypotheses, etc. More recently, the flexibility of notebook-style programming has been explored for providing code tutorials (Head et al., 2020).

Nevertheless, the above tools for notebooks assume developers produce the code, neglecting the support for code search activities that occur quite frequently in software development (Sadowski et al., 2015). Developers have to use generic search engines (e.g., Google Search) or those built for traditional code repositories (e.g., SourceForge), whose effectiveness is unknown due to the uniqueness of notebooks. In this paper, we focus on supporting code search for computational notebooks to improve developers’ efficiency in identifying desired code samples and APIs to use.

There exist several code search tools, in addition to generic search engines such as Google. Sim et al. (Sim et al., 2011) comprehensively compared Koders (Wikipedia contributors, 2020), Google, Google Code Search, Krugle (Krugler, 2013), and SourceForge (SourceForge, 2020), with different sizes of search target and motivation for search. These search engines index code based on API keywords and method signatures, which may have limitations because code cells in notebooks are organized quite freely and the signatures are difficult to extract. Other approaches have been explored to provide more semantic features, allowing developers to search more effectively. Examplar (McMillan et al., 2012) ranks search results using application descriptions, APIs used, and data-flows in code. CodeGenie (Lemos et al., 2007) leverages a test-driven approach to search reusable code. Sourcerer (Linstead et al., 2008) uses fine-grained structural information from the code, enabling capabilities that go beyond keyword-based search. However, none of the above search engines are designed to tailor searching code in computational notebooks, which have different characteristics compared to traditional code modules. For example, the code in notebooks is much messier, broken into small snippets (i.e., cells) that can be presented and executed in an arbitrary order. The analysis of code structures and semantics in existing tools relies on a clean, well-documented, and linearly-organized structure, which is often not available in notebooks. We approach this challenge by treating each cell as a search unit and identifying the relationships between cells within and across notebooks.

On the other hand, approaches for software visualization have been developed to understand the functional and structural components of existing code. For example, UML (Unified Modeling Language) is a standard diagram for showing relationships between software components. Hoffswell et al. (Hoffswell et al., 2018) provided an in-situ visualization within the code to summarize variable values and distributions. Based on a graph model, force-directed methods have been examined (Seider et al., 2016). Similarly, dependency graphs have been adopted to reveal connections between variables and functions (Balmas, 2004). Recently, Albireo (Wenskovitch et al., 2019) employs an interactive graph to visualize relationships of code and markdown cells in a notebook, while showing the information of variables, functions, and execution paths. TRACTUS (Subramanian et al., 2020) utilizes a tree structure to reveal the process of a hypothesis-driven analysis. While the above techniques are useful for understanding code in a single notebook, they cannot be easily applied for visualizing multiple notebooks, which is the case for browsing search results. NBSearch analyzes the relationships between cells within and between notebooks, presented in a novel visualization, NBLines. It also reveals the connections among variables and functions used in a notebook, helping developers make sense of search results and discover serendipitous opportunities.

It is important for users to comprehend search results, especially when they do not know the exact answer; they may try to gain an understanding of a topic to retrieve more specific or related information for subsequent search actions (Teevan et al., 2004). Wilson et al. (Wilson et al., 2010) advocated that the future web search interfaces should emphasize exploration. Users’ search behaviors in web visualizations have also been explored (Feng et al., 2018). There only exist a few works focusing on the sensemaking of code search results, many techniques have been proposed to visualize document and website searches or text.

One way of presenting search or recommended results is based on a linear ranked list like Google. TileBars (Hearst, 1995) places a colored bar next to each list item to display document length and term frequency. Moreover, PubCloud (Kuo et al., 2007) uses tag clouds to augment a ranked list. LineUp (Gratzl et al., 2013) allows users to visually manipulate a multi-attributed ranked list. uRank (di Sciascio et al., 2016) provides on-the-fly search results refinement and reorganization as information needs evolve.

On the other hand, researchers have utilized a 2D space to present documents with various layout methods. One approach is to place items based on their attributes (or facets); for example, PivotPath (Dörk et al., 2012) and PivotSlice (Zhao et al., 2013) display a publication corpus as a multi-attributed network and position each document interactively with a partition of a 2D canvas. Another approach is projection-based methods. Gomez-Nieto et al. (Gomez-Nieto et al., 2014) used an energy-based method to place text snippets of search results with minimal overlap. MOOCex (Zhao et al., 2018) visualizes a recommended list of videos on a 2D canvas using multidimensional scaling (MDS) with a specific similarity measure. Moreover, space-filling techniques (e.g., treemaps and 2D tiles) have been used for browsing searching results (Clarkson et al., 2009; Kleiman et al., 2015). Further, WebSearchViz (Nguyen and Zhang, 2006) and Topic-Relevance Map (Peltonen et al., 2017) employ a circular layout, where the distance to the center represents a document’s relevance, and the section of the circle denotes a specific topic or cluster. Likewise, RankSpiral (Spoerri, 2004) employs a spiral layout to render a ranked list by maximizing information density.

However, it is inadequate to directly apply the above methods in our scenario, because computational notebooks are different from text documents. For example, many of these techniques are based on topic analysis and text similarity, which cannot be extracted from code. Further, unlike documents or websites, the relationships (e.g., commonly-used variables and APIs) between code cells need to be revealed, but they have different distributions and characteristics compared to term relationships in documents.

We employed a participatory design approach to develop NBSearch, in which we actively involved two expert end-users over a six-month design process. Both experts work at Uber, which is a large IT company with more than 25K employees. One is a software engineer with more than three years of professional experience. The other is an engineering manager who has three years of development experience plus two years of management experience. They both use Jupyter Notebooks frequently every day to perform data analysis, conduct coding experiments, develop analytical models, etc. Moreover, both of them have once contributed to large open-sourced projects for Jupyter Notebooks. Both experts are also authors of this paper, for their contributions to the deployment of NBSearch using Uber’s internal data and server as well as participant recruitment (see Section 7.1 and Section 7.2). However, they were separated from the core research team for the design and development of NBSearch at the University of Waterloo.

Through the design process, we maintained bi-weekly meetings with the experts, during which they were presented with the latest ideas, designs, and prototypes for feedback on further improvements. We also set up a Slack channel to communicate irregularly for ad-hoc discussion, short consultation, and quick feedback. The whole design process consists of three stages: define & ideate (1.5 months), protype & test (3 months), and deploy & evaluate (1.5 months). During the first stage, we conducted a series of in-depth interview and brainstorming sessions with the experts. Together, we identified the common practices and seen problems of developers using Jupyter Notebooks in an enterprise setting. We thus iteratively distilled a set of design goals for guiding the development of code search systems with computational notebooks (see Section 4.2). We also drew a blueprint for the system with some level of technical consideration (see Section 4.3). In the second stage, a series of prototypes were developed and refined based on our experts’ feedback, including sketches, paper prototypes, analytical components with various machine learning models, and an interactive working prototype running on a public notebook corpus. During our meetings with the experts, we invited them to try out our prototypes and to comment on various aspects including visual design, interface layout, user interaction, search quality, etc. Figure 2 shows some early sketches and mockups for NBSearch, which set the foundation of our final design shown in Figure 1 (also see Section 6). Our development of the backend analytical pipelines was also supported by a set of micro-benchmarks and experiments (see Section 5). For the third stage, with the help of our experts, we deployed a fully-functional prototype of NBSearch on an internal notebook corpus from their company. We also carried out a user study to formally evaluate NBSearch as a whole with professional data scientists and software engineers at Uber (see Section 7).

As introduced earlier, we distilled the following set of design goals from the first stage of our design process.

G1: Retrieve relevant code segments with fuzzy search. Notebooks are designed to facilitate on-the-fly programming, in which developers need to constantly write and refine code to try different methods (Head et al., 2019). Code search activities with vague goals occur more often in such exploratory scenarios, and developers usually do not know what exactly they want to use (e.g., specific APIs) (Sim et al., 2011). Especially, our experts mentioned: “For new employees, it is infeasible to glance through all the notebooks due to the sheer volume. They often don’t know which keyword to use. This is where contextual search comes in handy.” Thus, the system needs to go beyond the traditional keyword-based search by supporting fuzzy search based on semantics. That is, a developer should be able to query in natural language by describing roughly what they need.

G2: Accommodate the flexibility of computational notebooks. “Seasoned developers often document and publish their workflow by leveraging notebooks’ dynamic nature for easy reproducibility and interpretability,” said by the experts. However, unlike traditional software code, each code cell in a notebook is relatively independent, and cells can be executed at an arbitrary order and for multiple times. This results in loosely connected code segments in notebooks and a highly nonlinear programming pattern for developers (Head et al., 2019; Subramanian et al., 2020; Wenskovitch et al., 2019). Classic code search by indexing functional modules may not be appropriate, because such concepts are unclear in notebooks. Hence, the system should be based on cells to adapt their flexibility and enable the discovery of “micro-components or steps in developers’ workflows.”

G3: Allow for exploring intra- and inter-notebook relationships. Due to the complex structure of notebooks, the results returned by a search engine, which contain code cells from multiple notebooks, can be complicated. Representing the results with a classic ranked list may not be sufficient (Zhao et al., 2018). Our experts also mentioned: “Existing platforms such as Google Search and StackOverflow do not help much because most of their software solutions are built in-house.” For each cell in the search result, code cells from the same notebook can be relevant to provide a context of the usage of the scripts in that cell; however, the cells before or after that cell may not be necessarily related because of the dynamic nature of notebooks. Further, given a specific cell of interest, code cells from other notebooks, including the cells returned in the search result and other cells co-existing in the associated notebooks (but not in the search result), can be useful for serendipitous discoveries. Therefore, the system should provide an effective means for helping developers explore such intra- and inter-notebook relationships to make sense of the search results.

Following these design goals, we developed NBSearch with the aforementioned iterative participatory design process with our experts. An overview of the system architecture is shown in Figure 3. Details of each component will be introduced in the following sections. Here we provide a high-level description of the system.

First, a search database needs to be built from a notebook corpus, as illustrated in the preparation & preprocessing step in Figure 3. For the notebook corpus, code cells were extracted and split into two sets: code with comments and code without comments. From the code cell with comments, a seq2seq translation model (e.g., an LSTM recurrent neural network (Hochreiter and Schmidhuber, 1997)) was trained to “translate” code scripts into corresponding comments, which we call text descriptors of the code. With the trained translation model, a completed database of code-descriptor pairs of all the code cells in the corpus was generated by running the model on the code cells without comments. Then, all the text descriptors were sent to a language model (e.g., Doc2Vec (Le and Mikolov, 2014)) to learn a semantic vector space of these texts. Thus, each of the descriptors could be represented by a numerical vector, stored in a database of descriptor vectors, where more similar descriptors have a closer distance in the vector space.

With these trained models and generated databases, NBSearch is able to handle a user’s query and perform a semantic search of code in the notebook corpus. As described in Section 3, when a user initiates a query in natural language on the search bar of the front-end interface, the back-end leverages the language model to convert this query into its vector form. Then, it uses this query vector to search in the formerly generated database of descriptor vectors and retrieve similar descriptors. Based on these descriptors, the system further looks up in the database of code-descriptor pairs to obtain the associated code cells. After, these code cells, along with other useful information (e.g., which notebooks they belong to, other cells within the same notebooks, variables/functions used in code), are sent to the front-end for search result presentation.

The semantic code search capability of NBSearch does not mean to replace the classic code keyword-based search. Users can initiate a pure keyword search by using quotation marks on keywords (e.g., “plot.hist”). This is useful when they know exactly what APIs to use and want to seek examples of how to use them. The keyword-based search was implemented with the Okapi BM25 method (Manning et al., 2008) by indexing the scripts of notebook code cells, where exact keyword matches are found and ranked by their frequency in code cells. However, NBSearch does not currently support semantic and keyword-based searches simultaneously in one single query, which is an interesting future direction.

We first need a corpus of computational notebooks to construct a database to support code search. We obtained data from UCSD digital collection (Collections, 2020), which contains 1.25M notebooks crawled from GitHub during 2017. As an initial attempt, in this paper, we used a sampled portion of the entire dataset for development and experimentation, which contains about 6K notebooks (around 10K code cells) as our corpus. However, our method can be directly employed on the entire UCSD notebook collection with larger and stronger computing resources.

To support semantic code search (G1), at a high level, we want to build a database maintaining a one-to-one mapping between code segments and text descriptors in natural language. Therefore, we can process a natural language query to search similar text descriptors, and thus find the associated code segments performing the described functions, procedures, etc. The code itself cannot be a meaningful semantic descriptor, because it is significantly different from natural language. Comments embedded in code are a straightforward source for constructing these descriptors, because they usually explain the underline semantic meaning of the code, among many other aspects such as control flows and desired outputs.

However, not every piece of code contains comments. Further, some comments are not in natural language (e.g., programmers comment out lines of code to compare different results). Thus, to utilize the code of the whole corpus, one approach is to generate the “comments” based on the code. Intuitively, we view it as a translation task in natural language processing, where we aim to build a model for translating programming language (e.g., Python code) into natural language (e.g., comments, or text descriptors), with the existing code and comments as the training data. With the trained model, we can generate descriptors from code that is without them, and thus build a complete dataset for all the code in the corpus (Figure 3).

As we consider notebook cells as the basic search unit (G2), we processed the notebook corpus by cells. We first scanned the entire corpus and extracted code cells with comments. Then, we concatenated all the comments in a code cell to form the text descriptors for this cell and stored pairs of code and descriptors as our training set. We did not consider markdown cells, although they are potential semantic descriptors for the code in a notebook. This is because it is challenging to identify which code cells are associated with which markdown cells. A markdown cell may describe multiple code cells in a notebook (which can be the cells immediately after or some other cells), or just generally describe the settings of the entire notebook (i.e., associated with no code). Due to this level of complexity, we decided not to use markdown cells since imprecise, noisy training data may affect the performance of the model. However, in the future, if there exists a high-quality labelled dataset showing the connections between markdowns and code in notebooks, our method can be easily applied and extended.

As mentioned above, we formulated our problem as a translation task in machine learning, i.e., from Python code (programming language) to text descriptors (natural language). With recent advances in neural machine translation, we employed the seq2seq model which is composed of an encoder and a decoder; both of them are recurrent neural networks (RNN) (Sherstinsky, 2018). We considered four different state-of-the-art RNNs in this study, including: GRU (gated recurrent unit) (Chung et al., 2014), LSTM (long short-term memory) (Hochreiter and Schmidhuber, 1997), LSTM with Attention (Luong et al., 2015), and BiLSTM (bidirectional LSTM) (Schuster and Paliwal, 1997). To assess their performance, we employed the BLEU (bilingual evaluation understudy) score (Papineni et al., 2002), which is a common measure of machine translation models. The results indicate that LSTM with Attention performed the worst, and the other three models performed similarly on our dataset. We thus manually browsed the results of GRU, LSTM and BiLSTM, and finally chose LSTM based on our observations and consultations with our experts. However, for deployment in practice, experiments might be required for applying the models on different datasets. Details about the data preprocessing, model training, and evaluation can be found in Section A.1.

Using the above trained translation model, we were able to generate descriptors for the code without sufficient comments. By combining the code cells with comments, we built a complete dataset of code-descriptors pairs for all the notebook code cells (Figure 3). However, having the descriptors for each cell is not enough to support semantic code search, as a user most likely input some queries that do not match the descriptors exactly. We need to characterize the similarity between the queries and the available descriptors in semantics (G1). Therefore, we leveraged neural language models in natural language processing, which could represent words, phrases, or sentences with vectors in a latent space. These vectors can be stored in the database, based on which we could retrieve semantically similar descriptors from an input query by simply computing their cosine similarity (Figure 3).

In this study, we considered two different language models: Doc2Vec (Le and Mikolov, 2014) and Sentence-BERT (Reimers and Gurevych, 2019). The goal of building the language model is to find descriptors that are semantically similar to a search query, however, this “semantically similar” criteria is very subjective. Unlike common language modeling tasks for English, there lacks ground truth or a standard benchmark to assess the performance of the models. While most of the comments are written in English words, the usage patterns and distributions may be different. Inspired by the literature (e.g., (Reimers and Gurevych, 2019)), we evaluated the two models using the downstream search tasks by conducting a survey study. We carefully selected 10 queries, such as “knn to perform classification” and “image manipulation.” We then generated 10 sets of search results for each of the models. In our survey study, we asked participants to select which ones of the 10 items are relevant in the search result, and then to indicate which search result of the two models is better. Based on the results, we concluded that Doc2Vec is more suitable for the language model with our dataset. However, more experiments might be needed when the models are deployed on another notebook corpus. Details about the model training, survey procedure, and results can be found in Section A.2.

As shown in Figure 1, the NBSearch user interface consists of four components: (A) the Search Bar, (B) the Search Result Panel, (C) the Notebook Relationship Panel, and (D) the Notebook Panel. These panels of NBSearch are interactively linked to many aspects, such as mouse hovering events. This interface was iterated several times with different layouts and design, as indicated in Figure 2ab.

After NBSearch is launched, only the Search Bar is shown, and a developer can configure the number of notebook cells to return (i.e., top-$k$ results where $k$ is 10 by default) at the top-right corner of the UI. We restricted that only one cell from a notebook can be included in the top-$k$ results, allowing for extending the search scope a bit. This is because some code cells within a notebook are very similar, because a developer sometimes tries different methods in exploratory programming by simply changing one or two parameters of the code (Wenskovitch et al., 2019). If there is no such restriction, the search results may contain several very similar code cells. In fact, the back-end search engine of NBSearch returns the whole notebooks containing the searched code cells to the front-end for display, and thus the information is not lost due to this restriction. This constraint can be turned off when needed.

After processing a query, the Search Result Panel (Figure 1B) appears to display a ranked list of searched code cells. Meanwhile, the Notebook Relationship Panel (Figure 1C) shows an interactive visualization—NBLines—of the searched code cells and the notebooks they belong to. The colors on both panels correspond to different notebooks; when there are more than 20 unique notebooks, the ones beyond the top 20 are colored in gray. Whenever a cell is clicked on the Search Result or Notebook Relationship Panels, the Notebook Panel (Figure 1D) shows the entire notebook containing that cell and jumps to that cell automatically. The colored bar on top of each cell indicates its similar with respect to the searched cell; so that the searched cell always has a full-length bar. Also, a series of tags indicate the variables and functions used in that cell, with dark gray tags showing the overlapping variables/functions between the selected cell and other cells. These features of the Notebook Panel help reveal the intra-notebook relationships.

The NBLines visualization displays both the intra- and inter-notebook cell relationships, allowing a developer to find connections between cells while maintaining a mental model of all notebooks. It contains two visual forms: the lines view and the dots view. In the lines view (Figure 4a), each notebook is visualized as a colored line, with cells represented as dots on the line, where the searched cell has a bigger size and a darker shade. The visual design was inspired by EgoLines (Zhao et al., 2016) that present relationships between actors in a temporal dynamic network. The cells across multiple notebooks are aligned based on their similarity. That is, each row, across the lines, contains cells that similar to each other; at the same time, the order of the cells in each notebook (i.e., columns) is kept. The order of the columns (or lines) is based on the search rank of the searched cell in that notebook, as indicated on Search Result Panel. The visualization reveals the inter-notebook relationship of cells, allowing a developer to look for similar cells across different notebooks, starting from the searched cell. This enables serendipitous findings in search results exploration.

When a cell is selected, several black curves appear to indicate the cells sharing more than $n$ variables/functions within the same notebook. The threshold $n$ can be adjusted interactively using a slider on top of the view. This reveals the intra-notebook relationship, allowing a developer to find relevant cells within a notebook. For example, in Figure 4b, the red arrow (on the right) indicates the cells linked by the black curves in the brown notebook all share the validation_data variable. It facilitates the sensemaking of a notebook’s structure, because the cells can be out of order.

The alignment of the notebook cells is very similar to the multiple sequence alignment (MSA) problem in computational biology, where several DNA or RNA sequences are aligned together to identify similar regions of nucleotide or amino acids in the sequences. MSA is an NP-hard problem, and a number of approximation algorithms have been proposed. In particular, we employed a progressive multiple sequence alignment algorithm (Higgins and Sharp, 1988) for its accuracy and speed. Such an approach progressively aligns multiple sequences by conducting pairwise sequence alignments in order and adjusting the global alignment. Our case is unlike the sequence alignment in computational biology, where the nucleotide or amino acids in different sequences need to be the same when aligned. We employed the cosine similarity between code cells to define the “optimal” alignment—having the smallest total similarity score for every pair of aligned cells in two sequences. The cell similarity can be easily computed because our search database contains the vector representation of all the cells. For example, in Figure 4a, the three red arrows (on the left) point to two exactly same (but from two different notebooks) cells (blue and red) and one similar cell (purple) on the same row, which all contains code about plotting.

The dots view of NBLines (Figure 4b) displays the same information as the lines view, using the same horizontal alignment and user interactions. However, cells are pushed to the left and each column may contain cells from multiple notebooks. This allows a developer to see the distribution of similar cells on all the rows. As shown in Figure 2cd, we also experimented with different visual designs for connecting the cells across notebooks in the dots view during the design process. In the end, we removed them because our experts thought the visuals were too overwhelming.

We recruited 12 participants, nine males and three females, aged 27–41 (four did not report it), through internal mailing lists at Uber with the help of our experts. They were all professional software engineers or data scientists with diverse backgrounds and from different engineering groups, covering a range of cases. Three of them have Bachelor’s, six have Master’s, and three have PhD degrees. The two experts involved in our design process were excluded from the participants pool. On a 1-7 Likert Scale, the medians of participants self-reported familiarity to Jupyter Notebooks and visualization techniques were 4.5 ($Mode=5$) and 4.5 ($Mode=3$), respectively (1 = “no experience at all” and 7 = “using almost everyday”). For the frequencies of conducting code search, six participants indicated they search code for 1-5 times daily, two for 6–10 times, two for 11–15 times, and two for over 20 times.

We rebuilt our back-search engine by retraining the translation and language models based on an internal notebook corpus at Uber, in order to best capture participants’ interests. The corpus contains 5,583 notebooks with around 365K code cells, where around 95K cells have comments. These notebooks were created for manipulating data, training and testing machine learning models, generating visualization dashboards, and some other tasks. We followed the same process to build the search engine using the same sets of parameters, as described in Section 5. We retrained the four translation models and found that GRU performed the best on this dataset (see Section A.3). Based on our previous experience (see Section 5.2) and detailed observations of the results, we chose Doc2Vec as the language model for our search engine in the study because of its better and stabler performance.

The study was conducted using a remote conferencing software, where participants used their own computers to access NBSearch deployed on the company’s internal server. During the study, participants first completed a brief training session to get familiar with the NBSearch interface by watching a short video.

Then, they were asked to conduct three free-form code search tasks under different predetermined scenarios, for example, “Imagine that you are writing some python code to plot your data with some charts, but you have no clue. Now please use the search engine to see how others write relevant code and find desirable results.” The search scenarios were carefully developed with our expert users who know the dataset and their daily activities very well. The goals were to prevent participants from searching stuff they are not interested or getting too few results due to the sparsity of the dataset. At a high level, the three task scenarios were about: how to train machine learning models (T1), how to plot charts from data (T2), and how to use visualization tools (T3). In each task, they were allowed to use either NBSearch’s semantic or keyword-based search capabilities, or a combination of both. There was no time limit for the search task, and participants were instructed to stop until they found what they needed in their minds. The order of the three tasks was counterbalanced across participants.

After, participants completed a post-study questionnaire about their impression of NBSearch, followed by a short semi-structured interview. Their search queries and interactions were logged, and the interview sessions were audio-recorded. The whole study lasted about 30 minutes for each participant.

In the following, we report our results from the user study, including participants’ performance in the search tasks, their questionnaire ratings, and their feedback to the system.

While our study results indicate that NBSearch can facilitate developers with code search during exploratory programming in large computational notebook repositories, it still has several limitations. We discuss them below and introduce potential solutions.

First, we did not involve markdown cells in building our search database of code-descriptor pairs, because it is hard to precisely identify which code cells are described by which markdown cell. Albireo (Wenskovitch et al., 2019) uses a naive method by simply associating a markdown cell with the code cell after. However, this oversimplifies the scenarios, given that notebooks are extremely flexible; sometimes a markdown cell may not correspond to any code in a notebook, and sometimes a markdown cell may describe a group of cells that are not necessarily the ones following it. It is very challenging to detect the relationships between code and markdowns using unsupervised methods, which needs a considerable amount of labelled data available in the future.

Second, currently NBSearch only enables semantic code search at the cell level, as the flexibility and organization of notebooks are based on cells; however, future extensions can be made to support search at sub-cell and multi-cell levels. For the sub-cell level, we can leverage the location of comments and the functional scope of code. For example, if a comment is inline with the start of a for-loop, we can pair all the code within the scope of the loop with this comment (or descriptors), similar for functions and statements. By enhancing the cell-level code-descriptor pairs with such information, the search engine is able to return finer or coarser-grained results based on a user’s query; and the analytical pipeline of NBSearch can naturally adapt to this new data without changes.

Third, the front-end interface and back-end models of NBSearch can be enhanced. In the controlled study, several participants mentioned that they spent quite some time to get familiar with the system interface, especially the NBLines, while they thought it was useful to explore the search results. They also thought the dots view was confusing. Future improvements may include removing the dots view and providing on-demand tutorials for the lines view. Also, NBSearch currently works as a standalone tool, but future development of NBSearch within the Jypyter Notebook eco-system will provide much convenience to users. This will also require setting up a public server for the search APIs. For the search engine, there were a couple of participants who were not very satisfied with the search results for the first few attempts. Trying other models or training them on a larger repository could resolve this issue; however, this may require humongous data with which the models can gradually become better, such as Google Search.

Finally, the evaluation of NBSearch can be further enhanced. Additional and larger testing datasets need to be used to thoroughly assess the performance of the translation model; and a large Mechanical Turk study, in a similar form to our survey study, may be conducted to deeply examine the quality of the language model. While we designed NBSearch through an iterative participatory design process with two experts and evaluated the system with professionals in the user study, a longer-term deployment is necessary for investigating the usefulness of NBSearch in practice. While there are currently no appropriate baselines, a comparative study between NBSearch and its base version (e.g., without NBLines and other visual cues) would provide deeper insights into the impact of the user interface. We leave these studies for future work.

We propose a general architecture to support semantic code search as illustrated in Figure 3. It is flexible enough to incorporate other translation and language models when more advanced ones are available in natural language processing research. Our test scripts also provide insights into which models to use for particular data in practice. As the business of each company may differ, retraining may be needed on their dataset for better retrieval; this is because different datasets may have different distributions and patterns of code usage. Our experts encountered no difficulty in retraining and deploying NBSearch with their dataset, which indicates the generalizability of our computational flow. We validated NBSearch with a large pool of professionals from Uber, supporting that our design can be generalized to various cases and needs. However, further validations in different cases need to be conducted.

Moreover, the NBLines visualization, equipped with a notebook alignment algorithm, provides an easy means of browsing the intra- and inter-notebook relationships of cells. It allows users to better understand the notebooks’ structures and discover serendipitous findings. This can be extended to show sub-cell level relationships when needed, e.g., by representing the functional unit of code as circles and cells as circle groups. It is useful for analyzing a collection of computational notebooks at a finer level, as sometimes code cells can be quite long depending on a developer’s habits. The visualization itself is general enough to apply on other problems involving the comparison of multiple sequences, such as clickstream data in web analytics.