Submodular Bandit Problem Under Multiple Constraints

1. 1.

   We propose a submodular bandit problem with semi-bandit feedback under the intersection of $l$ knapsacks and $k$-system constraints (Section 4). This is the first attempt to solve the submodular bandit problem under such complex constraints. The problem is new even when the $k$-system constraint is a cardinality constraint.

2. 2.

   We propose a novel algorithm called AFSM-UCB that Adaptively Focuses on a Standard or Modified Upper Confidence Bound (Section 5).

3. 3.

   We provide a high-probability upper bound of an approximation regret for AFSM-UCB (Section 6). We prove that the $\alpha$-approximation regret $\mathrm{Reg}_{\alpha}\left(T\right)$ is given by $O(\sqrt{mT}\ln(mT/\delta))$ with probability in least $1-\delta$ and the computational complexity in each round is given as $O(m|\mathcal{N}|\ln|\mathcal{N}|/\ln(1+\varepsilon))$, where $\alpha=\frac{1}{(1+\varepsilon)(k+2l+1)}$, $\varepsilon$ is a parameter of the algorithm, $T$ is the time horizon, $m$ is the cardinality of a maximal feasible solution, and $\mathcal{N}$ is the ground set (e.g., the set of all news articles in the news recommendation example). We note that no known offline fast²²2We refer to Section 2.1 for the meaning of “fast”. algorithm achieves a better approximation ratio than above³³3See footnote in page 1..

4. 4.

   We empirically prove the effectiveness of our proposed method by comprehensively evaluating it on a synthetic and two real-world datasets. We show that our proposed method outperforms the existing greedy baselines such as LSBGreedy and CGreedy.

Although submodular maximization has been studied over four decades, we introduce only recent results relevant to our work. Badanidiyuru and Vondrák [2014] provided a maximization algorithm for a non-negative, monotone submodular function with $l$ knapsack constraints and a $k$-system constraint that achieves $\frac{1}{(1+\varepsilon)(k+2l+1)}$-approximation solution. Based on this work and Gupta et al. [2010], Mirzasoleiman et al. [2016] proposed a maximization algorithm called FANTOM under the same constraint in the case when the objective function is not necessarily monotone. Our proposed method is inspired by these two offline algorithms. However, because of uncertainty due to semi-bandit feedback, we need a nontrivial modification. A key feature of our method and aforementioned two offline algorithms is that they filter out “bad” elements via a threshold. Such a threshold method is also used for other problem settings such as streaming submodular maximization under a cardinality constraint [Badanidiyuru et al., 2014]. Some algorithms [Sarpatwar et al., 2019, Chekuri et al., 2010, 2014] achieves better approximation ratios than that of [Badanidiyuru and Vondrák, 2014] under narrower classes of constraints (e.g., a matroid + $l$ knapsacks). However, these algorithms are not “fast” because their computational complexity is $O(\mathrm{poly}(|\mathcal{N}|))$ with a polynomial of high degree, while that of [Badanidiyuru and Vondrák, 2014] is $O(\frac{|\mathcal{N}|}{\varepsilon^{2}}\ln^{2}\frac{|\mathcal{N}|}{\varepsilon})$. For example, the computational complexity of the algorithm provided in [Sarpatwar et al., 2019] is $\widetilde{O}(|\mathcal{N}|^{6})$ when $k=1$. We refer to [Sarpatwar et al., 2019, Mirzasoleiman et al., 2016] for further comparison with respect to an approximation ratio and computational complexity.

Yue and Guestrin [2011] introduced the linear submodular bandit problem to solve a diversification problem in a retrieval system and proposed a greedy algorithm called LSBGreedy. Later, Yu et al. [2016] considered a variant of the problem, that is, the linear submodular bandit problem with a knapsack constraint and proposed two greedy algorithms called MCSGreedy and CGreedy. Chen et al. [2017] generalized the linear submodular bandit problem to an infinite dimensional case, i.e., in the case where the marginal gain of the score function belongs to a reproducing kernel Hilbert space (RKHS) and has a bounded norm in the space. Then, they proposed a greedy algorithm called SM-UCB. Recently, Hiranandani et al. [2019] studied a model combining linear submodular bandits with a cascading model [Craswell et al., 2008]. Strictly speaking, their objective function is not a submodular function. Table 1 shows a comparison with other submodular bandit problems with respect to constraints.

In this subsection, we define submodular functions. We refer to [Krause and Golovin, 2014] for an introduction to this subject.

We denote by $2^{\mathcal{N}}$ the set of subsets of $\mathcal{N}$. For $e\in\mathcal{N}$ and $S\subseteq\mathcal{N}$, we write $S+e=S\cup\left\{e\right\}$. Let $f:2^{\mathcal{N}}\rightarrow\mathbb{R}$ be a set function. We call $f$ a submodular function if $f$ satisfies $\Delta f(e|A)\geq\Delta f(e|B)$ for any $A,B\in 2^{\mathcal{N}}$ with $A\subseteq B$ and for any $e\in\mathcal{N}\setminus B$. Here, $\Delta f(e|A)$ is the marginal gain when $e$ is added to $A$ and defined as $f(A+e)-f(A)$. We note that a linear combination of submodular functions with non-negative coefficients is also submodular. A submodular function $f$ on $2^{\mathcal{N}}$ is called monotone if $f(B)\geq f(A)$ for any $A,B\in 2^{\mathcal{N}}$ with $A\subseteq B$. A set function $f$ on $2^{\mathcal{N}}$ is called non-negative if $f(S)\geq 0$ for any $S\subseteq\mathcal{N}$. Although non-monotone submodular functions have important applications [Mirzasoleiman et al., 2016], we consider only non-negative, monotone submodular functions in this study as in the preceding work [Yue and Guestrin, 2011, Yu et al., 2016, Chen et al., 2017].

Many useful and interesting functions satisfy submodularity. Examples include coverage functions, probabilistic coverage functions [El-Arini et al., 2009], entropy and mutual information [Krause and Guestrin, 2005] under an assumption and ROUGE [Lin and Bilmes, 2011].

For succinctness, we omit formal definitions of the matroid and $k$-system. Instead, we introduce examples of matroids and remark that the intersection of $k$ matroids is a $k$-system. For definitions of these notions, we refer to [Calinescu et al., 2011].

First, we provide an important example of a matroid. Let $\mathcal{N}_{i}\subseteq\mathcal{N}$ ($i=1,\dots,n$) be a partition of $\mathcal{N}$, that is $\mathcal{N}$ is the disjoint union of these subsets. For $1\leq i\leq n$, we fix a non-negative integer $d_{i}$ and let $\mathcal{P}=\left\{S\in 2^{\mathcal{N}}\mid|S\cap\mathcal{N}_{i}|\leq d_{i},\ \forall i\right\}$. Then, the pair $(\mathcal{N},\mathcal{P})$ is an example of a matroid and called a partition matroid. Let $d$ be a non-negative integer and put $\mathcal{U}=\left\{S\in 2^{\mathcal{N}}\mid|S|\leq d\right\}$. Then $(\mathcal{N},\mathcal{U})$ is a special case of partition matroids and called a uniform matroid. Let $(\mathcal{N},\mathcal{M}_{i})$ for $1\leq i\leq k$ be $k$ matroids, where $\mathcal{M}_{i}\subseteq 2^{\mathcal{N}}$. The intersection of matroids $(\mathcal{N},\cap_{i=1}^{k}\mathcal{M}_{i})$ is not necessarily a matroid but a $k$-system (or more specifically it is a $k$-extendible system) [Calinescu et al., 2011, Mestre, 2006, 2015]. In particular, any matroid is a $1$-system. For a $k$-system $(\mathcal{N},\mathcal{I})$ with $\mathcal{I}\subseteq 2^{\mathcal{N}}$ and a subset $S\subseteq\mathcal{N}$, we say that $S$ satisfies the $k$-system constraint if and only if $S\in\mathcal{I}$. Trivially, a uniform matroid constraint is equivalent to a cardinality constraint.

Next, we provide a definition of knapsack constraint. Let $c:\mathcal{N}\rightarrow\mathbb{R}_{>0}$ be a function. For $e\in\mathcal{N}$, we suppose $c(e)$ represents the cost of $e$. Let $b\in\mathbb{R}_{>0}$ be a budget and $S\subseteq\mathcal{N}$ a subset. We say that $S$ satisfies the knapsack constraint with the budget $b$ if $c(S):=\sum_{e\in S}c(e)\leq b$. Without loss of generality, it is sufficient to consider the unit budget case, i.e., $b=1$.

Following [Yue and Guestrin, 2011], we assume that there exist $d$ known submodular functions $f_{1},\dots,f_{d}$ on $2^{\mathcal{N}}$ that are linearly independent and the objective submodular function $f$ can be written as a linear combination $f=\sum_{i=1}^{d}w_{i}f_{i}$, where the coefficients $w_{1},\dots,w_{d}$ are non-negative and unknown to the algorithm. We fix a parameter $B>0$ and assume that $\sqrt{\sum_{i=1}^{d}w_{i}^{2}}\leq B$. We also assume that for some $A>0$, the $L^{2}$-norm of vector $[\Delta f_{i}(e\mid S)]_{i=1}^{d}$ is bounded above by $\sqrt{A}$ for any $e\in\mathcal{N}$ and $S\in 2^{\mathcal{N}}$.

We note that this can be generalized to an infinite dimensional case as in [Chen et al., 2017]. We discuss this setting more in detail in the supplemental material and provide a theoretical result in this setting.

We assume that there exists $m\in\mathbb{Z}_{>0}$ such that $m_{t}\leq m$ for all $t$ and consider the lexicographic order on the set $\left\{(t,i)\mid t=1,\allowbreak\dots,\allowbreak\ 1\leq i\leq m\right\}$, i.e., $(t,i)\leq(t^{\prime},i^{\prime})$ if and only if either $t<t^{\prime}$ or $t=t^{\prime}$ and $i\leq i^{\prime}$. Then, we can identify the set with the set of natural numbers (as ordered sets) and can regard $\{\varepsilon_{t}^{(i)}\}_{t,i}$ as a sequence. We assume that the stochastic process $\left\{\varepsilon_{t}^{(i)}\right\}_{t,i}$ is conditionally $R$-sub-Gaussian for a fixed constant $R\geq 0$, i.e., $\mathbf{E}\left[\exp\left(\xi\varepsilon_{t}^{(i)}\right)\mid\mathcal{F}_{t,i}\right]\leq\exp\left(\frac{\xi^{2}R^{2}}{2}\right),$ for any $(t,i)$ and any $\xi\in\mathbb{R}$. Here, $\mathcal{F}_{t,i}$ is the $\sigma$-algebra generated by $\left\{S_{u}^{(1:j)}\mid(u,j)<(t,i)\right\}$ and $\left\{\varepsilon_{u}^{(j)}\mid(u,j)<(t,i)\right\}$. This is a standard assumption on the noise sequence [Chowdhury and Gopalan, 2017, Abbasi-Yadkori et al., 2011]. For example, if $\{\varepsilon_{t}^{(i)}\}$ is a martingale difference sequence and $|\varepsilon_{t}^{(i)}|\leq R$ or $\{\varepsilon_{t}^{(i)}\}$ is conditionally Gaussian with zero mean and variance $R^{2}$, then the condition is satisfied [Lattimore and Szepesvári, 2019].

As usual in the combinatorial bandit problem, we evaluate bandit algorithms by a regret called $\alpha$-approximation regret (or $\alpha$-regret in short), where $\alpha\in(0,1)$. The approximation regret is necessary for meaningful evaluation. Even if the submodular function $f$ is completely known, it has been proved that no algorithm can achieve the optimal solution by evaluating $f$ in polynomial time [Nemhauser and Wolsey, 1978].

We denote by $OPT$ the optimal solution, i.e., $OPT=\operatorname{argmax}_{S}f(S),$ where $S$ runs over $2^{\mathcal{N}}$ satisfying the constraint (1). We define the $\alpha$-regret as follows:

This definition is slightly different from that given in [Yue and Guestrin, 2011] because our definition does not include noise as in [Chowdhury and Gopalan, 2017]. In either case, one can prove a similar upper bound. For the proof in the cardinality constraint case, we refer to Lemma 4 in the supplemental material of [Yue and Guestrin, 2011].

In this study, we take the same approximation ratio $\alpha=\frac{1}{(1+\varepsilon)(k+2l+1)}$ as that of a fast algorithm in the offline setting [Badanidiyuru and Vondrák, 2014, Theorem 6.1]. As mentioned in Section 2, there exist offline algorithms that achieve better approximation ratios than above, but they have high computational complexity. Later, we remark that our proposed method is also “fast”.

For $e\in\mathcal{N}$ and $S\in 2^{\mathcal{N}}$, we define a column vector $x(e\mid S)$ by $\left(\Delta f_{i}(e\mid S)\right)_{i=1}^{d}\in\mathbb{R}^{d}$ and put $x_{t}^{(i)}=x\left(e^{(i)}_{t}\mid S_{t}^{(1:i-1)}\right)$. Here, we use the same notation as in Section 4. We define $b_{t},w_{t}\in\mathbb{R}^{d}$ and $M_{t}\in\mathbb{R}^{d\times d}$ as follows:

Here, $\lambda>0$ is a parameter of the model and for a column vector $x\in\mathbb{R}^{d}$, we denote by $x\otimes x\in\mathbb{R}^{d\times d}$ the Kronecker product of $x$ and $x$.

For $e\in\mathcal{N}$ and $S\in 2^{\mathcal{N}}$, we define $\mu(e\mid S):=w_{t}\cdot x(e\mid S)$ and $\sigma(e\mid S):=\sqrt{x(e|S)^{\mathrm{T}}\hskip 2.0ptM_{t}^{-1}\hskip 2.0ptx(e|S)}.$ Then, we define a UCB score of the marginal gain by

and a modified UCB score by $\mathrm{ucb}_{t}(e\mid S)/c(e)$. Here,

and $c(e):=\sum_{j=1}^{l}c_{j}(e)$. It is well-known that $\mathrm{ucb}_{t}(e\mid\phi,S)$ is an upper confidence bound for $\Delta f_{\phi}(e\mid S)$. More precisely, we have the following result.

Proposition 1 follows from the proof of [Chowdhury and Gopalan, 2017, Theorem 2]. We note that this theorem is a more generalized result than the statement above (they do not assume that the objective function is linear but belongs to an RKHS). In the linear kernel case, an equivalent result to Proposition 1 was proved in [Abbasi-Yadkori et al., 2011].

We also define the UCB score for a list $S\allowbreak=\left\{e^{(1)},\dots,e^{(m)}\right\}$ by $\mathrm{ucb}_{t}(S)\allowbreak=\mu_{t-1}(S)+3\beta_{t-1}\sigma_{t-1}(S).$ Here $\mu_{t}(S)$ and $\sigma_{t}(S)$ are defined as $\sum_{i=1}^{m}\mu_{t}(e^{(i)}\mid S^{(1:i-1)})$ and $\sum_{i=1}^{m}\sigma_{t}(e^{(i)}\mid S^{(1:i-1)}),$ respectively. The factor $3$ in the definition of $\mathrm{ucb}_{t}(S)$ is due to a technical reason as clarified by the proof of Lemma $1$ in the supplemental material.

In this subsection, we propose a UCB-type algorithm for our problem. We call our proposed method AFSM-UCB and its pseudo code is outlined in Algorithm 2. Algorithm 2 calls a sub-algorithm called GM-UCB (an algorithm that Greedily selects elements with Modified UCB scores larger than a threshold, outlined in Algorithm 1). Algorithm 1 takes a threshold $\rho$ as a parameter and returns a list of elements satisfying the constraint 1. Algorithm 1 selects elements greedily from the elements whose modified UCB scores $\mathrm{ucb}_{t}(e\mid S)/c(e)$ and $\mathrm{ucb}_{t}(e\mid\emptyset)/c(e)$ are larger or equal to the threshold $\rho$. If the threshold $\rho$ is small, then this algorithm is almost the same as a greedy algorithm, such as LSBGreedy [Yue and Guestrin, 2011]. If the threshold $\rho$ is large, then the elements with large modified UCB scores will be selected. Thus, the threshold $\rho$ controls the importance of the standard and modified scores. The main algorithm 2 calls Algorithm 1 repeatedly by changing the threshold $\rho$ and returns a list with the largest UCB score. We prove that there exists a good list among these candidates lists.

As remarked before, Algorithm 2 is inspired by submodular maximization algorithms in the the offline setting [Badanidiyuru and Vondrák, 2014, Mirzasoleiman et al., 2016]. However, we need a nontrivial modification since the diminishing return property does not hold for $\mathrm{ucb}_{t}(e\mid S)$ unlike the marginal gain $\Delta f(e\mid S)$. We note that $\mathrm{ucb}_{t}(e\mid S)$ can be large not only when the estimated value of $\Delta f(e\mid S)$ is large but also if the uncertainty in adding $e$ to $S$ is high. Therefore, we need additional filter conditions to ensure that $e$ is a “good” element. Natural candidates for the condition are that $\mathrm{ucb}_{t}(e\mid S^{(1:i)})/c(e)\geq\rho$ for some indices $i$. In Algorithm 2, we require $\mathrm{ucb}_{t}(e\mid\emptyset)/c(e)\geq\rho$ in addition to $\mathrm{ucb}_{t}(e\mid S)/c(e)\geq\rho$.

In the algorithm, we introduce parameters $\nu$ and $\nu^{\prime}$. The parameter $\nu$ (resp. $\nu^{\prime}$) is used for defining the initial (resp. terminal) value of the threshold $\rho$. In the next section, for a theoretical guarantee, we assume that $\nu\leq\max_{e\in\mathcal{N}}f(\left\{e\right\})\leq\nu^{\prime}.$ If the upper bound of the reward is known, then we can take $\nu^{\prime}$ as the known upper bound. In practice, it is plausible that most users are interested in at least one item in the entire item set $\mathcal{N}$, which implies $\max_{e\in\mathcal{N}}f(\left\{e\right\})$ is not very small. In addition, the number of iterations in the while loop in Algorithm 2 is given by $O(\ln\left(\nu^{\prime}|\mathcal{N}|/\nu\right))$. Therefore, taking a very small $\nu$ does not increase the number of iterations as much.

We discuss the computational complexity of Algorithm 2 and that of existing methods. We consider a greedy algorithm by applying LSBGreedy to our problem; i.e., we consider a greedy algorithm that selects the element with the largest UCB score until the constraint is satisfied. By abuse of terminology, we call this algorithm LSBGreedy. Similarly, when we apply CGreedy (resp. MCSGreedy) to our problem, we also call this algorithm CGreedy (resp. MCSGreedy). In each round, the expected number of times to compute $\mathrm{ucb}_{t}(e\mid S)$ in Algorithm 2 is given by $O(m|\mathcal{N}|\ln(\nu^{\prime}|\mathcal{N}|/\nu)/\ln(1+\epsilon))$, while that of LSBGreedy is given by $O(m|\mathcal{N}|)$. The computational complexity of MCSGreedy and CGreedy is given as $O(|\mathcal{N}|^{3})$ and $O(m|\mathcal{N}|)$ respectively. Therefore, ignoring unimportant parameters, our algorithms incur an additional factor $\ln|\mathcal{N}|/\ln(1+\varepsilon)$ compared to that of LSBGreedy and CGreedy.

In the setting of [Yue and Guestrin, 2011, Yu et al., 2016], greedy methods have good theoretical properties. However, we show that for any $\alpha>0$, these greedy methods incur linear $\alpha$-regret in the worst case for our problem. We denote by $\mathrm{Reg}_{\alpha,\mathrm{MCS}}(T)$ and $\mathrm{Reg}_{\alpha,\mathrm{LSB}}(T)$ the $\alpha$-regret of MCSGreedy and that of LSBGreedy, respectively. Then the following proposition holds.

We provide the proof in the supplemental material.

We provide a sketch of the proof of Theorem 1 and provide a detailed and generalized proof in the supplemental material. Throughout the proof, we fix the event $\mathcal{F}$ on which the inequality in Proposition 1 holds.

We evaluate the solution $S_{t}$ by AFSM-UCB in each round $t$. The following is a key result for our proof of Theorem 1.

Using Proposition 3, we can bound the approximation regret above by the sum of uncertainty $\beta_{t-1}\sigma_{t-1}(S_{t})$. Because the algorithm selects $S_{t}$ and obtain feedbacks for $S_{t}$, the sum of uncertainty can be bounded above by a sub-linear function of $T$.

In this synthetic dataset, we assume $d=15$ and $|\mathcal{N}|=1000$. We define $x(e)$ and costs for a knapsack constraint in a similar manner in [Yu et al., 2016]. We sample each entry of $x(e)$ from two types of uniform distributions. We assume that for each item $e$, the number of genres that have high information coverage is limited to two. More precisely, we randomly select two indices of $x(e)$ and sample entries from $U(0.5,0.8)$ and sample other entries from $U(0.0,0.01)$. We generate 100 user preference vectors $w$ in a similar way to $x(e)$. We also sample the costs of items uniform randomly from $U(0.0,1.0)$. In this dataset, we consider the intersection of a cardinality constraint and a knapsack constraint. The result is shown in Figure 1.

We perform a similar experiment in [Mirzasoleiman et al., 2016] but with a semi-bandit feedback. In MovieLens100K, there are 943 users and 1682 movies. We take $\mathcal{N}$ as the set of 1682 movies in the dataset. There are $d=18$ genres in this dataset. First, we fill the ratings for all the user-item pairs using matrix factorization [Koren et al., 2009] and we normalized the ratings $r$ so that $r\in[0,1]$. For each movie $e\in\mathcal{N}$, we denote by $r_{e}\in[0,1]$ the mean of the ratings of the movie for all users. We define $P(g\mid e)=r_{e}/|\mathcal{G}_{e}|$ if $g\in\mathcal{G}_{e}$, otherwise we define $P(g\mid e)=0$. We normalize $P(g\mid e)$ as previously mentioned, because if $w_{i}=1$ for all $i$, then we have $P(\{e\})=r_{e}$.

We define a similar knapsack, cardinality, and matroid constraints to those of [Mirzasoleiman et al., 2016]. For $e\in\mathcal{N}$, the cost $c(e)$ is defined as $c(e)=F_{\mathrm{Beta}(10,2)}(r_{e})$, where $F_{\mathrm{Beta}(10,2)}$ is the cumulative distribution function of the $\mathrm{Beta}(10,2)$. For a budget $b\in\mathbb{R}_{>0}$, we consider a knapsack constraint $c(S)\leq b$. The beta distribution lets us differentiate the highly rated movies from those with lower ratings [Mirzasoleiman et al., 2016]. We generate 100 user preference vectors $w$ in a similar way to the news article recommendation example. In this dataset, we consider the following constraints on genres in addition to the knapsack $c(S)\leq b$ and cardinality $|S|\leq m$ constraints, There are $k$ genres in MovieLens100K, where $k=d=18$. For each genre $g$, we fix a non-negative integer $a$ and consider the constraint $|\left\{e\in S\mid e\text{ has genre }g\right\}|\leq a$ for $S\subseteq\mathcal{N}$. This can be regarded as a partition matroid constraint. Therefore, the intersection of the constraints for all genres is a $k$-system constraint. One can prove that the intersection of this $k$-system constraint and a cardinality constraint is also a $k$-system constraint. The results are displayed in Figure 2 in the case of the matroid limit $a=3$.

From the Million Song Dataset, we select 1000 most popular songs and 30 most popular genres. Thus, we have $|\mathcal{N}|=1000$ and $d=30$. For active 100 users, we compute $P_{g}(e)$ and user preference vector $w$ in almost the same way as $\overline{w}(e,g)$ and $\theta^{*}$ in [Hiranandani et al., 2019] respectively. They assume that a user likes a song $e$ if the user listened to the song at least five times, however, we assume that a user likes the song if the user listened to the song at least two times. We consider the intersection of a cardinality and a knapsack constraint $c(S)\leq b$. We define a cost $c$ for the knapsack constraint by the length (in seconds) of the song in the dataset. The costs represent the length of time spent by users before they decide to listen to the song and we assume that it is proportional to the length of the song ⁴⁴4We can also assume that users listen to the song and give feedbacks later.. The results are displayed in Figure 3. We do not show the performance of RANDOM in the figure since it achieves only very low rewards.

In Figures 1a, 2a, 3a, we plot the cumulative average rewards for each algorithm up to time step $T=100$. In Figures 1b, 2b, and, 3b (resp. 1c, 2c, and, 3c), we show the cumulative average rewards at the final round by changing the budget $b$ (resp. by changing the cardinality limit $m$) and fixing the cardinality limit $m$ (resp. fixing the budget $b$). These results shows that overall our proposed method outperforms the baselines. We note that Figure 3 shows different tendency as compared to other datasets since popular items in the Million Song Dataset have high information coverage for multiple genres and about 47 $\%$ of the items have low information coverage (less than 0.01) for all genres. Figures 1b, 2b, and 3b also show the results for the case when the budget is sufficiently large. This is the case when LSBGreedy performs well and our experimental results show that even in this case, our method have comparable performance to greedy algorithms. Moreover, Figures 1c, 2c, and 3c also show the results in the case when the cardinality constraints are sufficiently mild. In this case, CGreedy performs well since the constraints are almost same as a knapsack constraint. The experimental results show that our method tends to have better performance than that of CGreedy even in this case.

The linear model considered in the main article can be generalized to an infinite dimensional case [Chen et al., 2017]. We let $D:=\Phi\times 2^{\mathcal{N}}\times\mathcal{N}$ and define $\chi:D\rightarrow\mathbb{R}$ by $\chi((\phi,S,e))=\Delta f_{\phi}(e\mid S)$. We assume that there exists an RKHS (reproducing kernel Hilbert space) $\mathcal{H}$ on $D$ with a positive definite (or linear) kernel $\kappa$ and $\chi$ belongs to $\mathcal{H}$ and the norm $|\chi|_{\mathcal{H}}$ is bounded by $B>0$. We assume that $\kappa$ is continuos on $D\times D$. We also assume that $\kappa(x,x)\leq 1$ for any $x\in D$. If $\kappa(x,x)\leq c$, then our $\alpha$-regret would increase by a factor of $\sqrt{c}$.

As for noises, we consider the same assumption as in the main article.