Revelio: ML-Generated Debugging Queries for Distributed Systems

We enlisted 800 users on Amazon Mechanical Turk to interact with our testbed’s applications. Turk users, unaware of the injected faults, were asked to report their experiences under these faulty scenarios via multiple choice and free-form questions. In total, we simulated 85 different faults (per app) and collected an average of 10 user reports per fault. We paired this training data with the system logs from our testbed, and a set of debugging queries generated by us to mimic those expected in debugging reports.

We surveyed many recent papers and blog posts that document or measure bugs in production settings. Our survey includes major outages in large-scale services (e.g., Dropbox (dro, [n. d.]), Kubernetes (kub, [n. d.])), bugs in cloud services (e.g., Google (Sigelman et al., 2010), Facebook (Kaldor et al., 2017), Azure (Liu et al., 2019)), and experiences with open source systems (e.g., Cassandra, HDFS (Yuan et al., 2014)). Our survey revealed the following bug categories:

1. (1)

   System software and configuration faults.

   • $\bullet$

     Resource underprovisioning (Kaldor et al., 2017; kub, [n. d.]): In such bugs (e.g., at Facebook (Kaldor et al., 2017)), the containers or VMs running parts of a distributed system are allocated insufficient CPU, memory, disk, or network bandwidth.

   • $\bullet$

     Component failures (Fonseca et al., 2007; Yuan et al., 2014; bas, [n. d.]; fac, [n. d.]; dro, [n. d.]): Failures are common at scale, and can result from a faulty physical machine, a bug in the machine’s hypervisor, or an unduly small amount of memory being allocated to a particular component.

   • $\bullet$

     Subsystem misconfigurations (Yuan et al., 2014; fac, [n. d.]; Liu et al., 2019): Errors in the internal configuration files for a given subsystem are common, especially given complex interoperation with other subsystems. Examples include incorrect hostname mappings that result in improper traffic routing and poorly configured values for timeouts or maximum connection limits (Liu et al., 2019).

2. (2)

   Network faults.

   • $\bullet$

     Network congestion (Kaldor et al., 2017): Within data centers (Kaldor et al., 2017), queues build up at various network locations (e.g., virtual and physical switches) that connect subsystems, either due to temporarily increased application traffic (e.g., TCP incast (Chen et al., 2009)) or cross traffic.

   • $\bullet$

     Incorrect network configuration (Kaldor et al., 2017; Yuan et al., 2014; fac, [n. d.]): Network devices (e.g., firewalls, NATs, switches) between subsystems that communicate via RPCs may be incorrectly configured with forward/drop rules. This could cause unintended forwarding of packets to a destination or incorrect packet dropping.

3. (3)

   Application logic faults.

   • $\bullet$

     Bugs within subsystems (Sigelman et al., 2010; Qin et al., 2005; Lu et al., 2008; Arora et al., 2018; Lu et al., 2005; ins, [n. d.]; kub, [n. d.]): Bugs in application logic are prevalent in practice (Netravali and Mickens, 2019; Abuzaid et al., 2018), and can result in a wide range of system effects. For example, certain bugs arise from (accidentally) inverted branch conditions that trigger seemingly inconsistent behavior: an application may traverse an incorrect branch and display incorrect content or result in a program error. In contrast, certain code changes can trigger performance degradations, e.g., if unnecessary RPC calls are generated between microservices.

   • $\bullet$

     Incorrect data exchange formats and values (Liu et al., 2019): Particularly in microservice settings as in Azure services (Liu et al., 2019), bugs can arise if the RPC formats of the sender and receiver do not match. For instance, a change in the API exposed by one microservice could result in a bug if its callers are unaware of this change. Also included in this category are certificate or credential updates that have only been partially distributed (resulting in access control errors).

Revelio must overcome four key challenges to generate debugging queries. First, the model has to combine and relate diverse and seemingly disparate data inputs. Second, the output space of queries is highly structured, making it harder than standard multi-label classification where each label is independent (BakIr et al., 2007). This is because all debugging queries for a tool are drawn from the same language grammar, unlike opaque and independent labels. Third, the space of potential queries for a given input is large, requiring new techniques to scale to large distributed systems. Fourth, as per our study of production bugs (§2), the model must be able to generalize in a specific sense: if a fault occurs at one location during training and is debugged with a specific query, then, during testing, the model must predict the same query with a different parameter if the same fault occurs at a different location.

We now briefly describe how we handle each challenge before formally describing our model in §4

Assume the user report $R$ to be in the form of raw text and $L$ to be a vector obtained by concatenating ordered vectors for each feature (Figures 9 in §A) extracted from the system logs (e.g., time-windowed average, min queueing delay). Recall from §3.2 that rank ordering per feature in $L$ enables our model to learn about the order statistics of feature values across subsystems, rather than about numerical or ordinal values at specific subsystems (Figure 2). From here, a straightforward way of modeling ${\mathbf{P}}_{1}(T|R,L)$ would be to use a multi-label classifier with each template $T$ being a different label. However, as discussed in §3, query templates are structured and made up of smaller atomic components (e.g., IF, MAX, MEAN statements). In other words, the ASTs of many query templates share common subtrees. Therefore, simply treating each template as an independent output label is wasteful in terms of not sharing statistical strength.

Therefore, we adopt a different approach to modeling the output templates. In order to preserve the structural aspects in queries, we represent each template $T$ in the form of an abstract syntax tree (AST). Each node in the tree is an operator (e.g., SELECT) and the edges represent how the operators are composed together to form larger trees.

We use a Graph Convolutional Network (GCN) (Kipf and Welling, 2016) to construct a vector representation $v_{T}$ for each query template’s abstract syntax tree. The GCN updates each node’s vector representation in the AST by pooling information from all its neighbors and performs this process multiple times, allowing it to combine information from all nodes in the tree. The GCN outputs a vector for each node in the tree – we take the vector of the root node $v_{T}$ to represent the tree’s information. In parallel, we use a contextual text encoder (BERT) (Devlin et al., 2018) to convert the issue report $R$ into a vector $v_{R}$ and pass the log $L$ through a linear neural network layer to get a vector $v_{L}$. $v_{R}$ and $v_{L}$ are concatenated and fed through a non-linear layer followed by a linear layer to get a single vector $v_{S}$ representing the system state from both internal and external viewpoints. Finally, we use both $v_{S}$ and $v_{T}$ to obtain a measure for how likely the template $T$ is applicable to the debugging scenario $\langle R,L\rangle$ (i.e., the probability of $T$ given $R$ and $L$). We then search for a set of neural network parameters that maximize this score (S) or likelihood. The sequence of operations are summarized as:

where $[;]$ represents a concatenation of two or more vectors and $GCN(T)[root]$ represents indexing the output of the GCN to get the vector of the root node.

All of the above operations represent a continuous flow of information through a single deep neural network whose parameters $\theta$ can be trained through back-propagation and stochastic gradient descent (Goodfellow et al., 2016). We use the following maximization objective to learn the parameters:

Enumerating all trees $T^{\prime}$ is intractable, so we employ Noise Contrastive Estimation (NCE) (Gutmann and Hyvärinen, 2010) and draw $m=2$ negative samples to form each $T^{\prime}$ to approximate the objective.

Now that we have a method to pick a template $T^{*}$, we need to fill in the values for each blank $b$ in $T^{*}$.²²2For ease of exposition, we will assume filling in a single value, but our method can easily be used to fill in multiple values, one at a time. Each template implicitly specifies the type of subsystem (e.g., switches for a Marple query) that is relevant for the fault at hand. Thus, using the template, we first extract a list of all relevant subsystems from the system logs $L$. For each subsystem $u$ in this list, we have a feature vector $L_{u}$ which summarizes all of its logs (e.g., avg/max queue depth, packet count; see Table 4 for a full list). We also include ranking information $rank_{u}$ for each feature in $L_{u}$ (e.g., $u$’s rank in queue depth across all switches). Note that ranks embed the same information as ordering from §4.1; ordering is not possible here because each $L_{u}$ pertains to only a single subsystem. We use these features, along with a vector representation of the blank in the template (described below), to pick the most likely subsystem for the blank.

We feed the template $T^{*}$ (represented as an AST) through the same GCN module as in §4.1 and choose the vector representation for blank $b$ to be the output vector of its corresponding node in the tree. This allows us to represent the requirements of $b$ using the properties of its neighboring nodes in the AST. Our goal is to then pick the most suitable subsystem $u$ for the blank, and return the corresponding system identifier (e.g., IP address or port number). We use a similar set of operations to those in §4.1 to pick the most likely $u$ to fill $b$:

where $rank_{u}$ indicates the rank of subsystem $u$ in its subsystem’s logs $L$, based on the feature of interest (e.g., rank of a switch, across all switches, based on mean queue depth). We then use an objective similar to Eq. 4 to maximize ${\mathbf{P}}_{2}$ over ground truth data and learn the model parameters $\phi$.

Once each of the two models above have been trained, during inference, we find the combination of query template and query parameters that maximizes the probability that the resulting query would result from the given system state vector. This probability in turn is the product of the two probabilities predicted by each of our models ${\mathbf{P}}_{1}$ and ${\mathbf{P}}_{2}$ above.

We can also pick the top $k$ most relevant queries, rather than just the single most relevant one, using the ranking produced by the probabilities above. If $|T|\times|U|$ proves to be very large, we can approximate the above computation by considering only the top few templates according to ${\mathbf{P}}_{1}(T|R,L)$.

For all FFN layers in our ranking model, we use two linear layers, each with hidden size 300, along with ReLU non-linearity. The GCN also uses a hidden vector size of 300. We use the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.0001.

Our goal is to run each application in a distributed and controlled manner, in order to scale to large workloads and deployments, consider broad sets of realistic distributed debugging faults, and ultimately generate complete debugging datasets for Revelio.

One approach would be to run each application service on VM instances in the public cloud. However, public cloud offerings typically hide inter-instance network components (e.g., switches, firewalls) from users, precluding the use of in-network debugging tools (§8).

Instead, we opt for a local emulation approach in which we run each application subsystem (or service) in a different container on the same physical machine, and specify the network infrastructure and connectivity between them. To do this, we use Containernet (Peuster et al., 2016), an extension of Mininet (Lantz et al., 2010) that can coordinate Docker (doc, 2019) containers, each running on a dedicated core; we assign a separate core for network operation (i.e., P4 switch simulation). Our testbed can be scaled up to support higher throughput by making use of additional physical machines through distributed emulation (max, [n. d.]).

As illustrated in Figure 4, for each application, we configure its subsystem/service containers into a star topology.

At the center of the topology is a router which is responsible for layer 3 faults (e.g., firewall configuration errors). Each subsystem is connected to this router via two P4 programmable switches (Bosshart et al., 2014) using Mininet. We set routing rules to ensure that all subsystems are appropriately reachable. Finally, for external reachability, a NAT is used to connect the central router to the host machine’s Internet-reachable interface.

We integrate four debugging tools into our setup:

• $\bullet$

  Marple (Narayana et al., 2017) is a performance query language for network monitoring that uses SQL-like constructs (e.g., groupby, filter) to support queries that track 1) per-packet and per-switch queuing delays, and 2) user-defined aggregation functions across packets.

• $\bullet$

  tcpdump (tcp, 2019) is an end-host network stack inspector which analyzes all packets flowing through the host’s network interfaces, and supports querying in the form of packet content filtering (e.g., by hostname, checksum).

• $\bullet$

  Uber’s Jaeger (Uber, 2019) is an end-to-end distributed systems tracing system which follows the OpenTracing specification (OpenTracing, 2019). With Jaeger, developers embed tracepoints directly into their system source code to log custom state (e.g., variable values), and then aggregate tracepoint and timing information to understand how data values and control state flow across time and subsystems.

• $\bullet$

  Google’s cAdvisor (Google, 2019) profiles the resource utilization of individual containers, logging the following every 1 second: instantaneous CPU usage, memory usage, disk read/write throughput, and cumulative number of page faults.

§A.1.2 presents more details regarding the integration and usage of each tool. We note that these tools represent only part of the state-of-the-art for network and distributed systems monitoring; our setup is amenable to others (Kaldor et al., 2017; Fonseca et al., 2007; app, [n. d.]).

To create debugging data from realistic debugging scenarios, we created an automatic fault injection service. We note that our goal is not necessarily to match the system scale at which production faults were reported, but instead to evoke the user reports, system log patterns, and queries that correspond to the reported fault categories.

Our service is guided by our literature survey of production faults and our findings at Anon (§2). Specifically, we incorporate faults that cover all of the categories discussed in §2, and match the ratios across categories with the data from Anon (Table 1). These categories cover the observable performance (i.e., increased system response times) and functionality (i.e., missing or inconsistent page content, crashes) issues for the applications we consider. Table 10 (in §A) provides more detailed examples of the specific faults we inject. We note that our service can be easily extended to incorporate new bug types.

To extract system logs, user reports, and debugging queries from our testbed (§5), we conducted a large-scale data collection experiment on Amazon Mechanical Turk. For each application, we set up an EC2 instance per fault that we consider (Table 3). Each instance runs the entire testbed for that application, with the associated fault injected into it. All instances for an application were populated with the same content, which was generated using an application-provided script, e.g., Reddit content was scraped from the live site.

Our experiment supported only “master” Turk users, and each was only allowed to participate once per fault+application pair. Each user was assigned to a specific instance/fault at random, and was presented with a UI (Figure 12 in §A) that had an iframe pointing to the corresponding instance’s frontend web server. Users were asked to perform multiple tasks within each application, including loading the homepage, clicking on item pages or user profiles, adding comments, and adding items to their carts. Prior to the experiment, users were shown representative pages for each step (to ensure familiarity with the interfaces), and told about expected bug-free load times (4-5 sec in our setup).

For each task, users were asked to report performance and functionality issues into a form that included both multiple choice and free-form questions. During each user’s experiment, the standard system logs for each tool in our testbed (§A.1.2) were collected on the instance; Table 4 lists the collected metrics. We condense and featurize the time-series data for each metric using standard summary statistics such as min, max, average, and median. Once a user completed the experiment, their reports were paired with the associated system logs. We allowed up to 5 concurrent users per instance, and system logs reflect the interactions of all concurrent users. To complete our dataset, for each fault, we generate a debugging query with the appropriate tool that sufficiently highlights the root cause for the fault. This query is intended to represent the result of a past (successful) debugging experience. Table 3 summarizes our dataset, and Table 9 (in §A) lists example user reports.

Table 5 lists three representative examples. For instance, the first example pertains to a fault in which the Memcache subsystem is down for Reddit. The correct query is a Marple query which tracks packet counts at the switch directly connected to the failed subsystem; this query would highlight the lack of incoming/outgoing network traffic from Memcache. However, Revelio’s top-ranked query was a Jaeger query which hones in on a tracepoint that contains an error message noting the inability to connect to Memcache.

Similarly, in the second example, a Sock Shop subsystem is not provisioned sufficient CPU resources to handle the incoming traffic. The correct query was a cAdvisor query that explicitly tracked the container’s CPU usage, but Revelio’s top-ranked query used Jaeger to track the high residual function execution times for the microservice running in that container.

In cases where Revelio’s top-ranked query is not the correct query, Revelio most often ranked the correct query as second or third. Thus, by outputting a sequence of ranked debugging queries, Revelio can provide developers with significant context about a fault from multiple vantage points.

Our study involved 20 PhD students and postdoctoral researchers in systems and networking. All participants brought their own laptops, but debugging tasks were performed inside a provided VM for uniformity. Prior to the study, the authors delivered a 5-hour tutorial explaining the testbed and Sock Shop UI/code base; the study only involved Sock Shop to ease the developers’ ability to become intimately familiar with the application to debug. For each tool (Marple, Jaeger, cAdvisor, tcpdump, Revelio), we described its logs, query language, and interface. Developers were given 1 hour to experiment with the testbed and resolve any questions.

During the study, developers were presented with a series of six debugging scenarios: 2 in-network faults for routing errors and congestion (targeting Marple), 2 system configuration faults for resource underprovisioning and component failures (targeting cAdvisor), and 2 application logic faults for branch condition and RPC errors (targeting Jaeger); we exclude end-host network faults due to time constraints. For each fault type, developers were randomly assigned to debug one fault using only the testbed’s tools, and one also using Revelio. Ordering of the faults and tool assignments was randomized across participants to ensure a fair comparison.

For each fault, developers were presented with 1) a user report, 2) system logs for all testbed tools collected during the faulty run, and 3) the faulty testbed code. Developers were given 30 mins to diagnose each fault and provide a short qualitative description of the root cause. For example, a routing configuration error that disconnected Cassandra could be successfully reported as “Cassandra could not receive any network packets, leading to missing page content.” When a developer believed she had found the root cause, she informed the paper authors who verified its correctness. If incorrect, the developer was told to keep debugging until a correct diagnosis was generated, or 30 mins elapsed. Developers were unrestricted in their debugging methodologies, e.g., they were not required to use queries, though most did. Without Revelio, developers had to generate any query they wished to issue on their own; with Revelio, developers could generate queries or use the 5 suggested by Revelio.

The results of our developer study were promising and suggest that Revelio can be an effective addition to state-of-the-art debugging frameworks in terms of accelerating root cause diagnosis. Across all of the faults, Revelio increased the fraction of developers who could correctly diagnose the faults within the given time frame from 60% to 90%. Further, as shown in Figure 8, Revelio sped up the average root cause diagnosis time by 72% ($\sim$14 minutes) in cases where the developers were able to report the correct root cause.

After the study, we asked each developer qualitative questions about their experience with Revelio.

The most commonly reported benefit of Revelio was in shrinking the set of tools and queries that a developer had to consider. The primary gripe was with respect to Revelio’s UI, which is admittedly unpolished. Most importantly, the response to ”Would you prefer to use existing systems and networking debugging tools with Revelio?”, was “yes” for all 20 participants.

There exist dozens of powerful data logging and querying tools for distributed systems (Mace et al., 2015; Scott et al., 2016; Uber, 2019; Sigelman et al., 2010; Mace and Fonseca, 2018; Fonseca et al., 2007; Kaldor et al., 2017), networks (Handigol et al., 2014; Narayana et al., 2016, 2017; Scott et al., 2014; Moshref et al., [n. d.]; Tammana et al., 2015, 2016; tcp, 2019), and end-host stacks (gdb, 2018; Alpern et al., 2000; Netravali and Mickens, 2019; Feldman and Brown, 1988; Ko and Myers, 2008; Lienhard et al., 2008; Viennot et al., 2013; Agrawal et al., 1991; Gyimóthy et al., 1999; Korel and Laski, 1988; Netravali et al., 2016). Although each tool provides powerful data logging and querying capabilities from different vantage points, two limitations exist. First, these tools are not coordinated and lack context about system-wide debugging. Thus, the cognitive burden of deciding which tools to use, when, and how (e.g., parameters) falls on developers. Revelio interoperates with these tools and alleviates this burden by automatically predicting helpful debugging queries. Second, these tools ignore natural language inputs, including user reports characterizing external behavior. Our results (§6) and prior work (Potharaju et al., 2013; Govindan et al., 2016) show that these inputs can be a rich source of debugging insights.