Consistency and Coherence from Points of Contextual Similarity

We can expect that our measure given by Eq.2 is not only applicable in more situations, but also more robust. In Figure 1 we show how the correlations with expert scores depend on the BERT layers from which the embeddings $e$ are taken. We allow embeddings for the summary and the text to be taken from different layers. Our motivation for this is that (1) the summary and the text have different information density, and (2) we may get more insight at how robust the measures are. Figure 1 shows Kendall Tau-c correlations. Spearman correlations manifest similar patterns, see Appendix C.

Indeed, we observe that for the consistency ESTIME-soft is more tolerant to the choice of the layers, including the higher and more flat peak around the layers 21 (compared to ESTIME). For all qualities ESTIME-soft is wider across the layers. Compared to consistency, the fluency peak at high levels is more pronounced, this might be because the fluency is closer to the task of tokens prediction, which is the actual goal of the BERT training.

Coherence and relevance are the qualities that, on opposite, require long-range context for making a judgement. It is interesting therefore that ESTIME-soft (and ESTIME) correlates the best with both of these qualities away from the diagonals in Figure 1, when the embeddings for the summary tokens and for the text tokens come from different layers. The shape of these heatmaps suggests that something is happening at the middle of the transformer, and that embeddings from different layers may have complementary information, beneficial for finding the points of contextual similarity, and ultimately to the correlations.

The usefulness of ESTIME-soft Eq.2 depends on two factors: (1) successful finding of the points of contextual similarity $(e_{i}e^{\prime}_{\alpha})$, where we now allow $e_{i}$ and $e_{\alpha}^{\prime}$ to be picked up from different layers, and (2) a reliable relation between the context $e_{i}$ and the ’response’ $\phi_{i}$ (likewise between $e_{\beta}^{\prime}$ and $\phi_{\beta}^{\prime}$). We speculate that satisfying these two factors using a single transformer layer is easier for a ’local context’ quality - consistency or fluency, - but it is more difficult for coherence or relevance, for which a longer range context is important.

The coherence and relevance correlations with ESTIME in Figure 1 suggests that there may be a qualitative difference between the context-relevant information in the lower half and the upper half of the layers. Transformer layers were probed in Zhang et al. (2020); Vasilyev and Bohannon (2021a) by their usefulness for estimating qualities of a summary; the layers were also inspected by other kinds of ’probes’ in Jawahar et al. (2019); Voita et al. (2019); Liu et al. (2019); Kashyap et al. (2021); van Aken et al. (2019); Wallat et al. (2020); De Cao et al. (2020); Timkey and van Schijndel (2021); Turton et al. (2021); Geva et al. (2021); Yun et al. (2021); Fomicheva et al. (2021).

Since the points of contextual similarity are defined in Eqs.1 and 2 by dot-product of the embeddings of the masked tokens, we are curious about the evolution of such embeddings through the layers, regardless of the summarization task. The embedding norm is one interesting characteristic, as being solely responsible for the peak around the layer 21 for the correlations with consistency Vasilyev and Bohannon (2021a). In Figure 2 we present the norm of masked token embeddings across the layers, using the same annotated part of SummEval dataset: 100 texts and 1700 generated summaries.

Notice that we are using these texts and summaries simply as separate texts, and simply considering the embeddings of all masked tokens of the text.

We observe that there is only minor difference between considering the tokens of the text and the tokens of the summary. The embeddings of the text tokens have slightly higher norm than embeddings of the summary tokens, probably boosted by having more related context for each token. More importantly, the CLS token, having more long-range semantic information, gets a big peak of the norm exactly in the middle of the transformer, and a big dip and peak at higher layers.

In Figure 3 we consider how much does an embedding changes from one layer to the next, on average.

The figure shows the norm of the increment of a masked token embedding from one layer to the next. The change exactly in the middle of the transformer shows up even stronger.

A cosine between the embeddings of the same token in two neighbor layers is shown in Figure 4.

There are almost no changes in the CLS token across the layers, except the strong change in the middle layers 10-13, and then at high layers starting from layer 18. The common text tokens keep going through substantial transformation, remarkably with a stable pace - the cosign between the embeddings of neighbor layers is about 0.95, - and with much larger changes at the few special layers at which the CLS embedding also changes.

The evolution of the masked token embedding across the layers reflected in Figures 2, 3, 4 is not related to the summarization: this is how the embeddings change for a text or a summary (see also Appendix B). Looking in retrospect on the coherence in Figure 1, it seems natural to split the layers-by-layers square into four quadrants.

As we have seen in Figure 1, the coherence and relevance would get higher correlations with expert scores when the embeddings are picked up from different layers. The layers 18-21 for summary and around 8-11 for text would give a better correlations for coherence than in Table 1. We may speculate that the reason for this is that coherence and relevance are not as ’local’ as consistency and fluency, and need a more varied information for a judgement. When forced to use the points of similarity between the summary and the text, they do better with different layers.

In principle there should be a way to estimate the coherence and fluency of a summary without using the text, but having a text as a helpful pattern makes it easier. For better understanding of how the text helps, we can consider estimation of coherence by using the order of the points of similarity. Let us consider a correlation between the sequence of summary token locations $i=0,1,2,...,N$ and the sequence of the corresponding similarity locations $\operatorname*{argmax}_{\alpha}(e_{i}e^{\prime}_{\alpha})$ of the tokens in the text:

$$\tau\left((i)_{i=1}^{N},(\operatorname*{argmax}_{\alpha}(e_{i}e^{\prime}_{\alpha}))_{i=1}^{N}\right)$$

(3)

Equation 3, with $\tau$ being Kendall Tau-c correlation, estimates how well the summary keeps the order of the points of similar context. Figure 5 shows correlations of this measure with expert coherence scores.

This figure is very different from the coherence part of Figure 1. For once, the diagonal - using the same layers for the summary and the text - is fine again. The plateau of high correlations is wide. Using the same layer for the summary and the text, we get correlation with coherence 0.20 or 0.21 in the layers from 7 to 15, which are quite far from the top layers. These layers, however, would be in the top part of a base BERT.

The coherence estimate by Eq.3 involves only the points of similar context, and (unlike Eqs.1 and 2) does not deal with comparing the ’responses’. Such context happens to be better represented for coherence by the layers in the lower-middle part of the large BERT.

For a better insight into how the consistency is a more ’local’ quality than coherence, we will modify Kendall Tau correlation between two sequences by introducing a restriction on the distance between the elements of the first sequence. In applying such a modified Kendall correlation in Eq.3, we would consider only such pairs of summary tokens locations $i$ that are not further from each other than an (integer) interval $d$:

$$\tau_{d}\left((i)_{i=1}^{N},(\operatorname*{argmax}_{\alpha}(e_{i}e^{\prime}_{\alpha}))_{i=1}^{N}\right)$$

(4)

In Eq.4 we use the modified ’local’ Kendall Tau correlation $\tau_{d}$. We define $\tau_{d}$ as a simple Kendall Tau correlation between two sequences $X$ and $Y$, but with a restriction on the distance between the elements $X$. As usual, $\tau_{d}=\frac{n_{c}-n_{d}}{n_{t}}$, where $n_{c}$ is the number of concordant pairs, $n_{d}$ is the number of discordant pairs, and $n_{t}$ is the total number of pairs, but the counts include only the pairs $\{(X_{i},Y_{i}),(X_{j},Y_{j})\}$ for which $|X_{i}-X_{j}|<=d$. The concordant pair is a pair satisfying the condition $(X_{i}>X_{j}\;\textrm{and}\;Y_{i}>Y_{j})\;\textrm{or}\;(X_{i}<X_{j}\;\textrm{and}\;Y_{i}<Y_{j})$, otherwise the pair is discordant. Note that in our case (Eq.4) the distance is the same no matter whether it is defined between the $X$ elements or between the $X$ ranks.

Figure 6 shows the correlations of the measure given by Eq.4 with SummEval expert scores of coherence and consistency.

We can make several simple conclusions from this figure and its underlying data:

1. 1.

   Consistency is a more ’local’ quality than coherence: keeping order on shorter distances provides better correlations. For consistency the layer 21 provides the best correlations - exceeding 0.18 - at the distances from 1 to 15. For coherence the layer 8 provides the best correlations - exceeding 0.22 - at the distances from 19 to 33.

2. 2.

   There is a wide range of layers that are helpful both for coherence and for consistency. The best layers are at the lower half of the BERT for the coherence, and close to the top for the consistency.

3. 3.

   The verification of facts (responses) by Eq.2 or 1 is not important (and maybe not needed) for coherence: estimation of order of similarities by Eq.4 or 3 is sufficient. Naturally, for consistency the measures by Eq.2 or 1, based on a (simplistic) verification of facts, provide better correlations.