JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data

A Gaussian Process (GP) is defined as a set of random variables such that any finite subset of them follows a multivariate normal distribution. A GP can be used to define a prior distribution over the unknown function $f$, which can be converted to a posterior distribution once we observe additional data. Formally, a GP prior is defined by a mean function $\mu_{0}:{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\to\mathbb{R}$ and a valid kernel function $\kappa:{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\times{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\to\mathbb{R}$. A kernel function $\kappa$ is valid if it is symmetric and the Gram matrix $K$ is positive semi-definite. Intuitively, the entries of the kernel matrix $K_{i,j}=\kappa(x_{i},x_{j})$ measure the similarity between any two points $x_{i}$ and $x_{j}$. Given points $X=\{x_{1},x_{2},\dots,x_{n}\}$, the distribution of the function evaluations $\mathbf{f}=[f(x_{1}),f(x_{2}),\dots f(x_{n})]$ in a GP prior follows a normal distribution, such that $\mathbf{f}|X\sim\mathcal{N}(\mu_{0}(X),K(X,X))$ where $\mu_{0}(X)=[\mu_{0}(x_{1}),\mu_{0}(x_{2}),\dots\mu_{0}(x_{n})]$ and $K(X,X)$ is a covariance matrix. For simplicity, we will henceforth assume $\mu_{0}$ to be a zero mean function.

Given a training dataset $\mathcal{D}$, let $X_{\mathcal{D}}$ and $\mathbf{y}_{\mathcal{D}}$ denote the inputs and their noisy observations. Since the observation model is also assumed to be Gaussian, the posterior over $f$ at a test set of points $X^{\ast}$ will follow a multivariate normal distribution with the following mean and covariance:

Due to the inverse operation during posterior computation, standard GPs can be computationally prohibitive for modeling large datasets. We direct the reader to (Rasmussen 2003) for an overview on GPs.

Bayesian Optimization (BO) is a class of sequential algorithms for sample-efficient optimization of expensive black-box functions (Frazier 2018; Shahriari et al. 2015). A BO algorithm typically runs for a fixed number of rounds. At every round $t$, the algorithm selects a query point $x_{t}$ and observes a noisy function value $y_{t}$. To select $x_{t}$, the algorithm first infers the posterior distribution over functions $p(\mathbf{f}|\{(x_{i},y_{i})\}_{i=1}^{t-1})$ via a probabilistic model (e.g., Gaussian Processes). Thereafter, $x_{t}$ is chosen to optimize an uncertainty-aware acquisition function that balances exploration and exploitation. For example, a popular acquisition function is the Upper Confidence Bound (UCB) which prefers points that have high expected value (exploitation) and high uncertainty (exploration). With the new point $(x_{t},y_{t})$, the posterior distribution can be updated and the whole process is repeated in the next round.

At round $t$, we define the instantaneous regret as $r_{t}=f(x^{*})-f(x_{t})$ where $x^{\ast}$ is the global optima and $x_{t}$ maximizes the acquisition function. Similarly, we can define the cumulative regret at round $T$ as the sum of instantaneous regrets $R_{T}=\sum_{t=1}^{T}r_{t}$. A desired property of any BO algorithms is to be no-regret where the cumulative regret is sub-linear in $T$ as $T\to\infty$, i.e., $\lim_{T\to\infty}\nicefrac{{R_{T}}}{{T}}=0$.

Our focus setting in this work is a variant of BO, called Multi-Task Bayesian Optimization (MBO). Here, we assume $K>0$ auxiliary real-valued black-box functions $\{f_{1},\dots,f_{K}\}$, each having the same domain $\textstyle\chi$ as the target function $f$ (Swersky, Snoek, and Adams 2013; Springenberg et al. 2016). For each function $f_{k}$, we have an offline dataset $\mathcal{D}^{(k)}$ consisting of pairs of input points $x$ and the corresponding function evaluations $f_{k}(x)$. If these auxiliary functions are related to the target function, then we can transfer knowledge from the offline data $\mathcal{D}^{\text{aux}}=\mathcal{D}^{(1)}\cup\dots\cup\mathcal{D}^{(K)}$ to improve the sample-efficiency for optimizing $f$. In certain applications, we might also have access to offline data from $f$ itself. However, in practice, $f$ is typically expensive to query and its offline dataset $\mathcal{D}^{f}$ will be very small.

We discuss some prominent works in MBO that are most closely related to our proposed approach below. See Section Related Work for further discussion about other relevant work.

Multi-task BO (Swersky, Snoek, and Adams 2013) is an early approach that employs a custom kernel within a multi-task GP (Williams, Bonilla, and Chai 2007) to model the relationship between the auxiliary and target functions. Similar to standard GPs, multi-task GPs fail to scale for large offline datasets.

On the other hand, parametric models such as neural networks (NN), can effectively scale to larger datasets but do not defacto quantify uncertainty. Hybrid methods such as DNGO (Snoek et al. 2015) achieve scalability for (single task) BO through the use of a feed forward deep NN followed by Bayesian Linear Regression (BLR) (Bishop 2006). The NN is trained on the existing data via a simple regression loss (e.g, mean squared error). Once trained, the NN parameters are frozen and the output layer is replaced by BLR for the BO routine. For BLR, the computational complexity of posterior updates scales linearly with the size of the dataset. This step can be understood as applying a GP to the output features of the NN with a linear kernel (i.e. $\kappa(x_{i},x_{j})=h_{\phi}(x_{i})^{T}h_{\phi}(x_{j})$ where $h$ is the NN function with parameters $\phi$). For BLR, the computational complexity of posterior inference is linear w.r.t. the number of data points and thus DNGO can scale to large offline datasets.

MT-ABLR (Perrone et al. 2018) extends DNGO to multi task settings by training a single NN to learn a shared representation $h_{\phi}(x)$ followed by task-specific BLR layers (i.e. predicting $f_{1}(x),...,f_{K}(x)$, and $f(x)$ based on inputs). The learning objective corresponds to the maximization of sum of the marginal log-likelihoods for each task: $\sum_{t=1}^{K+1}p(\mathbf{y}_{t}|w_{t},h_{\phi}(X_{t}),\sigma_{t})$. The main task is included in the last index, $w_{t}$ is the Bayesian Linear layer weights for task $t$ with prior $p(w_{t})=\mathcal{N}(0,\sigma_{w_{t}}^{2}I)$, $\sigma_{t}$ and $\sigma_{w_{t}}$ are the hyper-prior parameters, and $(X_{t},\mathbf{y}_{t})$ is the observed data from task $t$. Learning $h_{\phi}(x)$ by directly maximizing the marginal likelihood improves the performance of DNGO while maintaining the computational scalability of its posterior inference in case of large offline data. However, both DNGO and ABLR have implicit assumptions on the existence of a feature space under which the target function can be expressed as a linear combination. This can be a restrictive assumption and furthermore, there is no guarantee that given finite data such feature space can be learned.

MT-BOHAMIANN (Springenberg et al. 2016) addresses the limited expressivity of prior approaches by employing Bayesian NNs to specify the posterior over $\mathbf{f}$ and feed the NN with input $x$ and additional learned task-specific embeddings $\psi(t)$ for task $t$. While allowing for a principled treatment of uncertainties, fully Bayesian NNs are computationally expensive to train and their performance depends on the approximation quality of stochastic gradient HMC methods used for posterior inference.

The regression model in JUMBO is composed of two GPs: a warm-GP and a cold-GP denoted by $\mathcal{GP}^{\text{warm}}$$(0,\kappa^{w})$ and $\mathcal{GP}^{\text{cold}}$$(0,\kappa^{c})$, respectively. As shown in Figure 1, both GPs are trained to model the target function $f$ but operate in different input spaces, as we describe next.

$\mathcal{GP}^{\text{warm}}$ (with hyperparameters $\theta_{w}$) operates on a feature representation of the input space $h_{\phi}(x)$ derived from the offline dataset $\mathcal{D}^{\text{aux}}$. To learn this feature space, we train a multi-headed feed-forward NN to minimize the mean squared error for each auxiliary task, akin to DNGO (Snoek et al. 2015). Thereafter, in contrast to both DNGO and ABLR, we do not train separate output BLR modules. Rather, we will directly train $\mathcal{GP}^{\text{warm}}$ on the output of the NN using the data acquired from the target function $f$. Note that for training $\mathcal{GP}^{\text{warm}}$, we can use any non-linear kernel, which results in an expressive posterior that allows for exact and tractable inference using closed-form expressions.

Additionally, we can encounter scenarios where some of the auxiliary functions are insufficient in reducing the uncertainty in inferring the target function. In such scenarios, relying solely on $\mathcal{GP}^{\text{warm}}$ can significantly hurt performance. Therefore, we additionally initialize $\mathcal{GP}^{\text{cold}}$ (with hyperparameters $\theta_{c}$) directly on the input space $\textstyle\chi$.

If we also have access to offline data from $f$ (i.e. $\mathcal{D}^{f}$), the hyperparameters of the warm and cold GPs can also be pre-trained jointly along with the neural network parameters. The overall pre-training objective is then given by:

where $\mathcal{L}^{\mathcal{GP}}(\cdot|\mathcal{D}^{f})$ denotes the negative marginal log-likelihood for the corresponding GP on $\mathcal{D}^{f}$.

Post the offline pre-training of the JUMBO’s regression model, we can use it for online data acquisition in a standard BO loop. The key design choice here is the acquisition function, which we describe next. At round $t$, let $\alpha_{t}^{\text{warm}}(x)$ and $\alpha_{t}^{\text{cold}}(x)$ be the single task acquisition function (e.g. UCB) of the warm and cold GPs, after observing $t-1$ data points, respectively.

Our guiding intuition for the acquisition function in JUMBO is that we are most interested in querying points which are scored highly by both acquisition functions. Ideally, we want to first sort points based on $\alpha^{\text{warm}}$ scores and then from the top choices select the ones with highest $\alpha^{\text{cold}}$ score. To realize this acquisition function on a continuous input domain, we define it as a convex combination of the individual acquisition functions by employing a dynamic interpolation coefficient $\lambda_{t}(x)\in[0,1]$. Formally,

In Eq. 2, By choosing $\lambda_{t}(x)$ to be close to $1$ for points with $\alpha_{t}^{\text{warm}}(x)\approx\max_{x}\alpha_{t}^{\text{warm}}(x)$, we can ensure to acquire points that have high acquisition scores as per both $\alpha_{t}^{\text{cold}}(x)$ and $\alpha_{t}^{\text{warm}}(x)$. Next, we will discuss some theoretical results that shed more light on the design of $\lambda_{t}(x)$.

Here, we will formally derive the regret bound for JUMBO and provide insights on the conditions under which JUMBO outperforms GP-UCB (Srinivas et al. 2010). For this analysis, we will use Upper Confidence Bound (UCB) as our acquisition function for the warm and cold GPs. To do so, we utilize the notion of Maximum Information Gain (MIG).

This quantity depends on kernel parameters and the set $\textstyle\chi$, and also serves as an important tool for characterizing the difficulty of a GP-bandit. For a given kernel, it can be shown that $\Psi_{n}({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})\propto\Pi({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})$ where $\Pi({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})=|{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}|$ for discrete and $\text{Vol}({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})$ for the continuous case (Srinivas et al. 2010). For example for Radial Basis kernel $\Psi_{n}([0,1]^{d})\in O(\log(n)^{d+1})$. For brevity, we focus on settings where $\textstyle\chi$ is discrete. Further results and analysis for the continuous case are deferred to Appendix A.

For GP-UCB (Srinivas et al. 2010), it has been shown that for any $\delta\in(0,1)$, if $f\sim\mathcal{GP}(0,\kappa)$ (i.e., the GP assigns non-zero probability to the target function $f$), then the cumulative regret $R_{T}$ after $T$ rounds will be bounded with probability at least $1-\delta$:

with $C=\frac{8}{\log(1+\sigma_{n}^{-2})}$ and $\beta_{T}=2\log\left(\frac{|{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}|\pi^{2}T^{2}}{6\delta}\right)$.

Recall that $h_{\phi}:{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\to\mathcal{Z}$ is a mapping from input space $\textstyle\chi$ to the feature space $\mathcal{Z}$. We will further make the following modeling assumptions to ensure that the target black-box function $f$ is a sample from both the cold and warm GPs.

Based on the regret bound in Eq. 4, we can conclude that if the partitioning ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}$ is chosen such that $\Pi(\bar{\mathcal{Z}}_{g})\ll\Pi(\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g})$ and $\Pi({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g})\ll\Pi({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})$, then JUMBO has a tighter bound than GP-UCB. The first condition implies that the second term in Eq. 4 is negligible and intuitively means that $\mathcal{GP}^{\text{warm}}$ will only need a few samples to infer the posterior of $f$ defined on $\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g}$, making BO more sample efficient. The second condition implies that the $\Psi_{T-s}({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g})\ll\Psi_{T}({\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}})$ which in turn makes the regret bound of JUMBO tighter than GP-UCB. Note that ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}$ cannot be made arbitrarily small, since $\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g}$ (and therefore $\bar{\mathcal{Z}}_{g}$) will get larger which conflicts with the first condition.

Figure 2 provides an illustrative example. If the learned feature space $h_{\phi^{\ast}}(x)$ compresses set $\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g}$ to a smaller set $\bar{\mathcal{Z}}_{g}$, then $\mathcal{GP}^{\text{warm}}$ can infer the posterior of $g(h_{\phi^{\ast}}(x))$ with only a few samples in $\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g}$ (because MIG is lower). Such $h_{\phi^{\ast}}(x)$ will likely emerge when tasks share high-level features with one another. In the appendix, we have included an empirical analysis to show that $\mathcal{GP}^{\text{warm}}$is indeed operating on a compressed space $\mathcal{Z}$. Consequently, if ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}$ is reflective of promising regions consisting of near-optimal points i.e. ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}=\{x\in{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\mid f(x^{\ast})-f(x)\leq l_{f}\}$ for some $l_{f}>0$, BO will be able to quickly discard points from subset $\bar{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}}_{g}$ and acquire most of its points from ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}$.

The above discussion suggests that the partitioning ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}$ should ideally consist of near-optimal points. In practice, we do not know $f$ and hence, we rely on our surrogate model to define ${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{g}^{(t)}=\{x\in{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}\mid\alpha_{t}^{\text{warm}\ast}-\alpha_{t}^{\text{warm}}(x)\leq l_{\alpha}\}$. Here, $\alpha_{t}^{\text{warm}\ast}$ is the optimal value of $\alpha_{t}^{\text{warm}}(x)$ and the acquisition threshold $l_{\alpha}>0$ is a hyper-parameter used for defining near-optimal points w.r.t. $\alpha_{t}^{\text{warm}}(x)$. At one extreme, $l_{\alpha}\to\infty$ corresponds to the case where $\alpha_{t}(x)=\alpha_{t}^{\text{cold}}(x)$ (i.e. the GP-UCB routine) and the other extreme $l_{\alpha}\to 0$ corresponds to case with $\alpha_{t}(x)=\alpha_{t}^{\text{warm}}(x)$.

Figure 4 illustrates a synthetic 1D example on how JUMBO obtains the next query point. Figure 4(a) shows the main objective $f(x)$ (red) and the auxiliary task $f_{1}(x)$ (blue). They share a periodic structure but have different optimums. Figure 4(b) shows the correlation between the two.

Applying GP-UCB (Srinivas et al. 2010) will require a considerable amount of samples to learn the periodic structure and the optimal solution. However in JUMBO, as shown in Figure 4(c), the warm-GP, trained on ($h_{\phi^{\ast}}(x),y$) samples, can learn the periodic structure using only 6 samples, while the posterior of the cold-GP has not yet learned this structure.

It can also be noted from Figure 4(c) that JUMBO’s acquisition function is $\alpha_{t}^{\text{cold}}(x)$ when the value of $\alpha_{t}^{\text{warm}}(x)$ is close to $\alpha_{t}^{\text{warm}\ast}$. Therefore, the next query point (marked with a star) has a high score based on both acquisition functions. We summarize JUMBO in Algorithm 1.

The objective we wish to minimize is the validation error of a regression task after 100 epochs of training. For this purpose, we consider an offline dataset that consists of validation errors for some randomly chosen configurations after 3 epochs on a given dataset. The target task is to optimize this error after 100 epochs. In (Klein and Hutter 2019), the authors show that this problem is non-trivial as there is small correlation between epochs 3 and 100 for top-1% configurations across all datasets of interest.

Next, we consider a real-world use case in optimizing circuit design configurations for a suitable performance metric, e.g., power, noise, etc. In practice, designers are interested in performing layout simulations for measuring the performance metric on any design configuration. These simulations are however expensive to run; designers instead often turn to schematic simulations which return inexpensive proxy metrics correlated with the target metric.

In this problem, the circuit configurations are represented by an 8 dimensional vector, with elements taking continuous values between 0 and 1. The offline dataset consists of 1000 pairs of circuit configurations and 3 auxiliary signals including a scalar goodness score based on the schematic simulations. We consider the same baselines as before. We also consider BOX-GP-UCB (Perrone et al. 2019) which confines the search space to a hyper-cube over the promising region based on all auxiliary tasks in the offline data. Unlike the considered HPO problems, the offline circuit dataset contains data from more than just one auxiliary task, allowing us to consider BOX-GP-UCB as a viable baseline. MF-GP-UCB was ran with the schematic score as the lower fidelity approximation of the target function. We ran each algorithm with $l_{\alpha}=0.1$ for 100 iterations and measured simple regret against iteration. As reflected in the regret curves in Figure 7(a), JUMBO outperforms other algorithms.

Effect of auxiliary tasks. It is important to analyze how learning on other tasks affects the performance. To this end, we considered the circuit design problem with 1 and 3 auxiliary offline tasks. In Figure 7(b), task 1 (yellow) is the most correlated and task 3 (red) is the least correlated task with the objective function. The regret curves suggest that the performance would be poor if the correlation between tasks is low. Moreover, the features pre-trained on the combination of all three tasks provide more information to the warm-GP than those pre-trained only on one of the tasks.

BLR with JUMBO’s acquisition function. A key difference between JUMBO and ABLR (Perrone et al. 2018) is replacing the BLR layer with a GP. To show the merits of having a GP, we ran an experiment on Protein dataset and replaced the GP with a BLR in JUMBO’s procedure. Figure 7(c) shows that JUMBO with $\mathcal{GP}^{\text{warm}}$significantly outperforms JUMBO with a BLR layer.