PoBRL: Optimizing Multi-Document Summarization by
Blending Reinforcement Learning Policies

We leverage MMR to navigate the complicated and overlapping sentence space in the multi-document. MMR provides a balance between relevance and diversity. Formally, MMR is defined as:

where $S$ is the produced summary, $s_{i}$ is the $i^{th}$ sentence from the documents $R$ excluding $S$, and $\lambda$ is a parameter balancing between importance and diversity. In general, importance and redundancy functions can be in any form. In practice, eqn 1 is implemented as an iterative and greedy algorithm that loops through each sentence in the documents for a pre-defined summary length. The algorithm is shown in Appendix Section A.

As noted by McDonald (2007)Lin and Bilmes (2010), such iterative and greedy approach is non-optimal because the decision made at each time step is local and doesn’t consider its downstream consequences. The sequential nature of algorithm naturally fits within the reinforcement learning framework in which each sentence extraction action is optimized for future(downstream) cumulative expected reward.

Formally, we view the sentence extraction as a Markov Decision Process (MDP). We define a state $X_{t}=\{R,S\}$, which is a combination of input documents and extracted set of sentences at time $t$ respectively. At each time step, our policy makes a decision $a_{t}$ determining which sentence $s_{t}$ is to be extracted. We define our policy $\pi$ that searches over possible sentences across the space. Instead of having a pre-determined length of summary, we let our policy $\pi$ to optimize it directly. We define actions of two types:

With this action setup, our extractive agent can design the optimal strategy as when to stop extraction. When it instead decides to continue extracting sentences, each sentence in the input will be extracted according to the probability $P_{\pi}(\cdot|X_{t})$, where $X_{t}$ is the state which is used to keep track of what has been extracted so far ($S$), and what has not be extracted ($R$). Our goal then is to come up with an optimal extraction policy $\pi^{*}=\text{argmax}_{\pi\in\Pi}E(\sum_{t}r_{t}|X_{t},a_{t})$, where $r_{t}$ is the reward contribution of adding sentence $s_{t}$ onto the system summary $S$.

If we define the reward contribution to be the ROUGE score Lin (2004) that we wish to optimize, then it can be shown (see Appendix B) that :

Fig.1 shows the sentence search process for the multi-document space. For each ground truth sentence (labeled as blue), there might exist more than one candidate sentences coming across multiple related text. For the optimal summary $S^{*}$, we want to extract the single best sentence (colored dark orange) from a pool of similar looking sentences (region denoted as"relevant sentence space"in the figure).

For effective navigation in such a search space, we proposed our PoBRL framework(Algorithm 1). Based on eqn 2, our framework decomposes the importance and redundancy objectives independently into two sub-problems. Since each problem has its own local objectives, to maximize eqn 2, we simply design policies $\pi^{\text{imp}}$ and $\pi^{\text{red}}$ to optimize for each objective independently. At a high level, we can think of $\pi^{\text{imp}}$ helps the policy to narrow down the search by identifying the most relevant(or important) regions. From these regions, $\pi^{\text{red}}$ searches for the most relevant and diversify (non-redundant) sentence. This search procedure is graphically illustrated in Fig. 1 where $\pi^{\text{imp}}$ leads the search toward the region containing candidates (1,2,3) which are sentences relevant to the ground-truth sentence; then in this region, $\pi^{\text{red}}$ leads the search to discover the most relevant and non-redundant sentence (denoted as candidate 3*). Connecting $\pi^{\text{red}}$ and $\pi^{\text{imp}}$ together formulates the optimal policy $\pi^{\text{PoBRL}}$ which selects the most optimal sentence.

To generate labels, we utilize the following technique. For each sentence $gold_{k}$ in gold-summary, we find its most similar sentence from the input text. We define similarity using the ROUGE-L measure. That is, $\text{ label}_{k}=\text{argmax}\text{ ROUGE-L}(gold_{k},s_{k})$, for each $s_{k}$ in the example. We define the rewards in corresponding to the agent’s actions. Formally,

where $\text{summ}_{\text{sys}}$ is the system summary and $\text{summ}_{\text{gold}}$ is the gold summary. We utilize actor-criticKonda and Tsitsiklis (2003) to train our policy network. The gradient update for the weights of policy network will be:

where $\pi_{\psi}$($\cdot$) is the policy function that takes an input state and outputs an action (in our case, which sentence to extract), $\psi$ consists of the learnable parameters of the neural network. Next, we have

is the advantage function which measures the advantage of extracting a particular sentence $s_{t}$ over the rest of the sentences in the pool. And $V(X_{t})=E_{\pi_{\theta}}\{\sum\gamma^{t}r_{t+1}|X_{t}\}$ measures the quality of state, $Q(X_{t},a_{t}=s_{t})=E_{\pi_{\theta}}\{\sum\gamma^{t}r_{t+1}|X_{t},a_{t}=s_{t}\}$.

We instantiate two sentence extraction policy networks as in Algorithm 1, each with its own objective, and trained with actor-critic.

The formulation in eqn.4 and eqn.5 are good as long as we have a reasonably good $\lambda$. In practice, the $\lambda$ parameter can be a fixed numerical value that can be computed offline and determined beforehand. In reality, this is not an easy task because: 1) the $\lambda$ parameter might vary from document to document; and 2) the optimal value of $\lambda$ might change throughout the extraction process.

To solve this problem, we propose an innovative idea that requires no separate offline training process and can be computed online in an adaptive fashion by leveraging the power of the RL framework. When we train the network with actor critic, we get the advantage function automatically as a byproduct. Therefore, the advantage function from eqn 8 provides a quality measure by comparing sentences from the pool. Utilizing the advantage function, we can have a score function that rates each sentence in accordance with the importance and redundancy/diversity balance. Formally,

Our model deals with each input document by reading one sentence at a time. Each input sentence will first be encoded by the sentence encoder. We use a temporal-CNN to encode each sentence, and denoted each sentence encoded hidden state with $s^{art=j}_{enc_{i}}$, where $art=j$ indicates the sentence that came from the $j_{th}$ article and $i$ represents the sentence number from that article. Each of the encoded sentences will then go through two paths: the article-level path and the sentence aggregation path.

In our multi-document setting, each input contains one or more articles. Here, we are looking to formulate a representation for each article of the multi-document input. The article level path represents the dark blue colored region in Fig 2. We implement the article-level encoder with a bidirectional LSTM. For each article of the multi-document input, this encoder takes its input sentences (eg, $s^{art=1}_{enc_{1}}$, $s^{art=1}_{enc_{2}},....s^{art=1}_{enc_{n}}$). This article is represented by the corresponding article level sentence encoded hidden states: $h^{art=j}_{i}=\text{LSTM}(s^{art=j}_{enc_{i}},h^{art=j}_{i-1})$ for $i\in[1,n]$. We repeat this process for all the article of the multi-document input text.

In this path, we feed in each sentence $s^{art=j}_{enc_{i}}$ for $i\in[1,n],j\in[1,m]$ one by one by concatenating them together as if they came from a single piece of text. We utilize a pointer network Vinyals et al. (2015) that captures local contextual and semantic information. This pointer network utilizes information by the article level encoder and form the basis for sentence extraction. We implement the pointer network with encoder-decoder based LSTM. Formally, we can write:

where $c_{t}$ is the output of the attention mechanism at each output time $t$, $e_{k}$ is the hidden state of the encoder pointer network LSTM. In other words, ($e_{1}$, $e_{2}$, $e_{3}$, …, $e_{k}$, …, $e_{n\times m}$) are the encoder hidden states for the input sentences $s^{art=j}_{enc_{i}}$ for $i\in[1,n],j\in[1,m]$. Lastly, $v,W_{sent}$, and $W_{art}$ are the network learn-able weight matrices.

The attention module integrates article level information to the sentence aggregate level extraction network.We implement this module with dot product attention Luong et al. (2015).

Let $d_{t}$ to be hidden state of decoder pointer network LSTM at output time index $t$, $h_{k}$ to be the corresponding article level sentence encoded hidden state, then:

and that $\text{score}(d_{t},h_{k})=d_{t}^{T}h_{k}$.

Now, we can connect the dots between each component above, and this builds our sentence extractor. During the actual sentence extraction process, we ranked each candidates in the pool set by: $P(s_{k}|s_{k-1},...s_{0})=\text{softmax}(u_{k}),\text{for }k\in\text{pool set}.$ The pool set is initiated to contain all of the sentences in the input text. When a particular sentence has been extracted, we remove it from the pool set. That is: $\text{Pool Set }\leftarrow\text{Pool Set }\setminus\{s_{k}\}$

Training a reinforcement learning neural network from scratch is difficult and time consuming because the initial policy network tends to behave randomly. To accelerate this learning process, we warm start it with supervised learning.

Warm Start: Supervised Learning We modify the training objective to be negative log-likelihood, and we train our network by maximizing this objective function. After pre-trained supervised learning, we can then train with actor-critics.

We report the ROUGE $F_{1}$ score on with ROUGE-1, ROUGE-2, ROUGE-SU(skip bigrams with a maximum distance of four). For our PoBRL model, we instantiated two actor-critic policy networks representing $\pi^{\text{imp}}_{\theta},\pi^{\text{red}}_{\phi}$ respectively. We then blend these two polcies by forming $\pi^{\text{PoBRL}}$ as in eqn.4 with extraction probability in eqn.5. For complete network hyper-parameter and training details, please refer to Appendix D.

Besides using a fixed $\lambda$ value, we also evaluated a $\lambda$ value calculated using the advantage function as in eqn.9, and denote it by PoBRL($\lambda=\lambda_{t}^{\text{adv}}$). At each extraction step, the optimal $\lambda$ is recalculated by balancing between the importance and redundancy consideration.

We empirically evaluate the performance of the models on the following two multi-document datasets.

Multi-News The Multi-News Fabbri et al. (2019) is a large dataset that contains news articles scrapped over 1500 sites. The average number of words per doc and ground-truth summaries are much longer than DUC-04. This dataset has 44972 number of training examples, 5622 for validation and test set respectively. We trained our models on this dataset, and reported their performance in Table 1. Our best model PoBRL ($\lambda$ = $\lambda_{t}^{\text{adv}}$) achieves a score of 46.51/17.33/42.42, outperforms other baselines in all metrics (with the strongest ones being MatchSum and MGSum). Here, we also show the performance of our model on different values of $\lambda$. When $\lambda=1.0$, the model reduces to single policy and is not policy-blended (denoted by RL w/o Blend). When $\lambda$ = $\lambda_{t}^{\text{adv}}$, the value of $\lambda$ is adaptively determined by the advantage function as in eqn. 9. For a fixed value of $\lambda=0.8$, we achieve a good score (45.13/16.69/41.12). However, it is intuitive that the optimal value should be varied from document to document, and it might change during the extraction process. That is reflected by the PoBRL($\lambda=\lambda_{t}^{\text{adv}}$) advantage method which provides a significant performance boost. On the other hand, the RL w/o Blend represents a model without policy blend strategy. Comparing RL w/o Blend with PoBRL demonstrates the effect of our policy-blending algorithm. Incorporating policy-blending on top of this hierarchical sentence extraction model provides remarkable performance boost (going from RL w/o Blend’s 44.01/15.65/33.96 to PoBRL’s 46.51/17.33/42.42).

DUC-04 This is a test only dataset that contains 50 clusters with each cluster having 10 documents. For each cluster, the dataset provides 4 human hand-crafted summaries as ground-truth. Following Fabbri et al. (2019)Lebanoff et al. (2018), we trained our models using the CNN-DM Chen et al. (2016) dataset, and report the performance of different models on Table 3. Our best model (PoBRL $\lambda$ = $\lambda_{t}^{\text{adv}}$) achieves 38.67/10.23/13.19 (ROUGE-1/2/SU) which is a substantial improvement over all other baselines (with the closest ones being OCCAMS-V and GRU+GCN). On the other hand, using a fixed value of $\lambda=0.8$ of PoBRL still achieves good performance (38.13/9.99/12.85). Comparing PoBRL’s $\lambda=0.8$ to $\lambda=\lambda_{t}^{\text{adv}}$ shows the power of our method of calculating $\lambda$ with the advantage function. The effect of our policy blending algorithm is demonstrated by comparing RL w/o Blend with PoBRL. Here, we also observe a major performance boost (36.95/9.12/11.66 versus 38.67/10.23/13.19).

In this experiment, we randomly sampled 100 summaries of Multi-News dataset from the following models: Hi-Map, CopyTransformer, MatchSum, and PoBRL. Following Chu and Liu (2019), we asked judges²²2All judges are native English speakers with at least a bachelor’s degree and experience in scientific research. We compensated the judges at an hourly rate of $28. (10 for each sample) to rate the quality of system generated system in a numerical rating of 1 to 5, with the lowest score to be worst and the highest score to be the best. We evaluated the quality of the system generated summarization in terms of the following metrics as in Chu and Liu (2019) Dang (2006): Grammatically, Non-redundancy, Referential clarity, Focus, Structure and Coherence. Our result is shown in Table 2.

Unlike DUC04, each example of the multi-news dataset has different number of source documents (range from 2 to 9). In general, the higher the number of the example, the longer the input. In Fig 3, we show the effect of our proposed method in handling different numbers of source document in each multi-doc input. As can be seen in the figure, the performance is almost uniformly distributed, showing that our method can handle higher or lower number of input documents (which also translate to longer or shorter piece of input text) equally well.

We measure the redundancy of our system summary by comparing the policy blending formulation of our model (i.e., PoBRL) against with our non-policy-blend (i.e., RL w/o Blend) model. This comparison allows us to understand the effectiveness of employing the policy-blending strategy. We measure redundancy by making use of the ROUGE-L score and the BERT’s Devlin et al. (2019) embedding, separately. Our evaluation criterion is the following: for each sentence in the generated summary, we compute the redundancy metric between that particular sentence against with the rest of the sentence in the summary. In ROUGE-L experiment, we report average sentence level ROUGE-L score within the input example. PoBRL reports the lowest score of 0.19 versus 2.28 as in RL w/o Blend, signifying the significant reduction in overlap (or redundancy) in the system summary. In the BERT’s experiment, we report the average cosine similarity score. As shown in Table 4, the PoBRL model achieves higher cosine measure (signifies more diversify and less repetitive).