Text Fact Transfer

Text style transfer aims to modify the style of text without inherently affecting its content Fu et al. (2018); Jin et al. (2022); Hu et al. (2022). The concept of style can take many forms, including formality Wang et al. (2019); Zhang et al. (2020), sentiment Prabhumoye et al. (2018); Yang et al. (2018), and authorship Xu et al. (2012). Text fact transfer is the counterpart to text style transfer, as we focus on transferring the factual content of text between topics without affecting its underlying style. Hence, our task emphasizes generating new, factual text, which is not the main focus of style transfer tasks.

Several methods have been developed for text style transfer, such as training neural models on parallel corpora Rao and Tetreault (2018); Xu et al. (2019), latently disentangling content and style Hu et al. (2017); Shen et al. (2017), or prototype editing Li et al. (2018); Sudhakar et al. (2019); Abu Sheikha and Inkpen (2011). ModQGA is most similar to the Delete-Retrieve-Generate model Li et al. (2018), which extracts attribute markers, transfers attributes across styles, and generates an output. We apply a similar technique for text fact transfer, but notably use a novel combination of end-to-end question generation and specificity-aware question answering, which has not been explored in prior work.

Recent work has studied models that leverage stylistic exemplars to guide stylistic choices in text generation Cao et al. (2018); Wei et al. (2020). Such exemplars improve the fluency of seq2seq models in various tasks, including summarization Dou et al. (2021); An et al. (2021), machine translation Shang et al. (2021); Nguyen et al. (2022), dialogue generation Zheng et al. (2020); Wang et al. (2021), and question answering Wang et al. (2022).

More relevant to text fact transfer are tasks that require strictly adhering to the style of an exemplar. Chen et al. (2019) propose the task of controllable paraphrase generation, which aims to combine the semantics from one sentence and the syntax from a second sentence. Lin et al. (2020) introduce “style imitation” and perform data-to-text generation while strictly maintaining the style of an exemplar. Apart from a lack of focus on transferring factual content, these works differ from our task as they do not leverage a factual corpus.

The task most similar to ours is expository text generation (ETG) Balepur et al. (2023), which seeks to generate factual text from a corpus in a consistent style. However, ETG dictates that this style is learned from examples of outputs in the same domain, while text fact transfer adheres to the style of a single source text. Hence, an ETG model is domain-specific, while a single text fact transfer model (e.g., 0-shot ModQGA) could be used in several domains. Further, the IRP model proposed by Balepur et al. (2023) for ETG combines content planning, retrieval, and rephrasing, while ModQGA modifies a source text with question generation and answering, and our model tackles the additional problem of controlling specificity (section 3.3).

The concept of transferring entities between topics is similar to analogy completion Ushio et al. (2021); Bhavya et al. (2022); Chen et al. (2022), which aims to select a word that parallels an input query-word pair (e.g., “Paris:France, Lima:[MASK]”). While analogy completion could be used for factual entity transfer, this is only one aspect of text fact transfer. Further, our task is fundamentally a text generation task, while analogy completion is typically used to assess how models internally capture relations.

Our approach to text fact transfer is rooted in the observation that all entities in the source text that need to be transferred can be viewed as an answer to a question. For example, given the source text “Ibuprofen is used to relieve pain,” transferring between the topics Ibuprofen and Melatonin may result in the text “Melatonin is used to promote sleep.” The part of the source text that needs to be transferred (apart from the known transfer of “Ibuprofen” to “Melatonin”) is “to relieve pain,” which can answer the question “Why is Ibuprofen used?”. This question-answer paradigm helps us guide the modification process in text fact transfer.

Hence, to identify entities that need to be transferred and the questions that can be answered by said entities, we train an end-to-end question generation model that jointly generates entities and their questions from a context. To do so, we leverage the SQuAD-V2 dataset Rajpurkar et al. (2016). We train BART-large Lewis et al. (2020a) to minimize the loss $\lambda_{qg}$ of token prediction of question $q$ and answer (entity) $e$, surrounded by <|question|> and <|answer|> tokens (represented as $\langle q\cdot e\rangle$), conditioned on the context $c$ and topic $t$:

If a generated question $q$ contains the source topic $t_{s}$, $q$ can be simply transferred to the target topic $t_{t}$ by replacing $t_{s}$ with $t_{t}$. To elicit this desirable property, we only keep SQuAD entries where the topic is a substring of the question.³³3We found that performing lexically constrained token decoding Hokamp and Liu (2017) with topic $t_{t}$ led to a similar outcome, but this resulted in higher GPU memory usage. Thus, all training questions contain the topic $t_{t}$, teaching the model to produce $t_{t}$ in the output during inference.

To ensure all factual entities in the source text are detected, we use nucleus sampling to generate $n$ question/entity pairs $\mathcal{Q}_{s}=\{(q_{j},e_{j})\}_{j=1}^{n}$ with each sentence of the source text $\mathcal{D}_{s}$ as the context $c$ and source topic $t_{s}$ as the topic $t$. Thus, each unique factual entity from the source text may be mapped to multiple questions, which are ensembled by the specificity-aware question answering model (section 3.3).

As a final post-processing step, we discard pairs in $\mathcal{Q}_{s}$ with an entity that does not appear in the source text, as this entity is hallucinated. Further, if an entity is a substring of another entity in $\mathcal{Q}_{s}$, we discard the substring (shorter) entity.

Each question $q_{j}$ found in the question/entity pairs $(q_{j},e_{j})\in\mathcal{Q}_{s}$ pertain to the source topic $t_{s}$, but we require a transferred question $q_{j}^{\prime}$ that pertains to the target topic $t_{t}$. Since $q_{j}$ will contain the substring $t_{s}$, we can simply replace $t_{s}$ with $t_{t}$ to obtain $q_{t}^{\prime}$.

Through testing on our validation sets, we also find that generic queries can outperform specific queries when retrieving contexts for question answering. For instance, we find that the generic query “What is the hub of the airport?” outperforms the specific query “What is the hub of Cathay Pacific Airport?”. We find this occurs because the inclusion of the topic $t_{t}$ in the query distracts the retriever when searching $\mathcal{C}$ for contexts in QA, as it is more biased towards facts that contain the tokens in $t_{t}$, even when said facts are not relevant to the query.⁴⁴4We note that the specific query can find the facts with top-$k$ retrieval for large $k$, but our QA model is trained to use fewer contexts ($k=5$), hence our need for generic queries. We show the benefit of generic queries experimentally with ablation studies (section 5.3).

To obtain a generic question $q_{j}^{\prime\prime}$, we take the intersecting tokens of $q_{j}$ and $q_{j}^{\prime}$, which eliminates the topic-specific tokens found in $t_{s}$ and $t_{t}$. Combining the specific and generic questions, we obtain a set of transferred questions and their corresponding source entities $\mathcal{Q}_{t}=\{(q_{j}^{\prime},e_{j})\}_{j=1}^{n}\cup\{(q_{j}^{\prime\prime},e_{j})\}_{j=1}^{n}$.

After creating the transferred question-entity pairs $\mathcal{Q}_{t}$, we faithfully answer each transferred question by retrieving a context from the factual source $\mathcal{C}$ followed by extractive question answering (QA). However, an off-the-shelf QA model cannot be used for our task, as it fails to consider the specificity of the answer we require Huang et al. (2022). To extract transferred entities that are stylistically aligned with the source text, we seek answers with the same level of specificity as the entities they are replacing. For example, at one step in ModQGA, we may obtain the question “Where is Stanford located?” derived from the source entity “rural”. While “California,” “Palo Alto,” and “suburban” are all valid answers, “suburban” is the best choice, as it shares the same level of specificity as “rural,” and thus best matches the style of the source text.

To create a dataset with these specifications, we again modify the SQuAD-V2 dataset Rajpurkar et al. (2016). The dataset already provides questions, contexts, and answer spans, but we still require guidance to match the specificity levels of the answers. We find that one way to obtain specificity guidance of an answer is through the skip-gram assumption Mikolov et al. (2013)—similar words are discussed in similar contexts. For example, we intuit that because “rural” and “suburban” are used in the same context (e.g., “the location is [suburban/rural]”), they have similar specificity levels. Hence, we obtain specificity guidance for each answer in the SQuAD dataset by replacing every word in the answer with a random top-20 skip-gram synonym via fastText embeddings Joulin et al. (2017).

For each transferred question/source text entity pair $(q,e)\in\mathcal{Q}_{t}$, we first use Contriever Izacard et al. (2022) to obtain the context $c$, i.e., the top-$k$ most relevant facts to $q$ in $\mathcal{C}$ via maximum inner-product search Shrivastava and Li (2014). Next, the question $q$, entity $e$ (specificity guidance), and context $c$ are fed through the specificity-aware QA model $p(\langle a_{i},a_{j}\rangle|c,q,e)$. We record the predicted answer $a=\langle a_{i},a_{j}\rangle$ with the highest likelihood (sum of start and end likelihoods) under length $m$.

We map each unique entity $e$ in $\mathcal{Q}_{t}$ to the answer $a$ with the highest total likelihood. This process returns a map $\mathcal{E}$ with each source text entity $e$ as the key and its transferred entity $a$ as the value.

Lastly, we infill the source text $\mathcal{D}_{s}$, replacing each entity $e$ with its mapped entity $a$ in $\mathcal{E}$. We describe zero-shot and supervised infilling methods below:

Although ModQGA is designed primarily as a 0-shot text fact transfer model, using custom components trained on external SQuAD datasets, altering the infilling process allows us to fairly compare our model with supervised baselines (section 4.2).

In Algorithm 1, we use the above components to design ModQGA. First, ModQGA performs end-to-end question generation with the BART model $p(\mathcal{Q}_{s}|\mathcal{D}_{s},t_{s})$, to generate $n$ question/entity pairs $\mathcal{Q}_{s}$ covering the factual content of the source text $\mathcal{D}_{s}$. ModQGA then transfers the questions in $\mathcal{Q}_{s}$ from the source topic $t_{s}$ to the target topic $t_{t}$ to create $\mathcal{Q}_{t}$, which has specific and generic questions. For each transferred question $q$ and source entity $e$ in $\mathcal{Q}_{t}$, ModQGA performs specificity-aware QA with the BERT model $p(\langle a_{i},a_{j}\rangle|c,q,e)$. We build the map $\mathcal{E}$, containing each source entity $e$ mapped to the answer $a$ with the highest likelihood, to represent its transferred entity. Last, using $\mathcal{E}$, ModQGA infills the source text $\mathcal{D}_{s}$ to create the target text $\mathcal{D}_{t}$, either in a 0-shot or supervised manner.

We adapt the following tasks and datasets to construct parallel corpora for text fact transfer:
1) Expository Text Generation (ETG) uses topic-related sentences to create multi-sentence factual texts in a consistent style Balepur et al. (2023). We adapt the U.S. News and Medline datasets, spanning college and medical domains. We use the output text as the target text and retrieve/create training examples for the source text (see Appendix A.1). We use the document titles for the source/target topics, and the provided corpus for $\mathcal{C}$.
2) Relationship triples have a subject $x$, predicate $y$, and relation $r$ between $x$ and $y$. We adapt the t-REX Elsahar et al. (2018) and Google Orr (2013); Petroni et al. (2019) relationship triple datasets. t-REX contains open-domain relations, while Google contains biographical relations. The open-domain nature of t-REX allows us to assess the adaptability of each baseline. We obtain triples that share a relation $r$ (i.e., $\langle x_{1},r,y_{1}\rangle$ and $\langle x_{2},r,y_{2}\rangle$) and use $x_{1}\cdot r\cdot y_{1}$ as the source text and $x_{2}\cdot r\cdot y_{2}$ as the target text ($\cdot$ denotes concatenation). We use $x_{1}$ and $x_{2}$ as the source and target topics. For t-REX, we use the Wikipedia texts in the dataset for $\mathcal{C}$, and for Google, we scrape sentences from the top-7 web pages queried with $x_{2}$.

We compare zero-shot ModQGA (0-shot ModQGA) with the following zero-shot baselines:
1) 0-Shot GPT: We use a zero-shot prompt (Appendix A.3) instructing GPT-3.5 to create the target text using the source text, source topic, and target topic. This model uses its internal knowledge.
2) 0-Shot GPT+Retr: We add the top-5 retrieved facts from $\mathcal{C}$ as an extra input to 0-Shot GPT.
3) SourceCopy: We trivially copy the source text as the predicted output for the target text.

When parallel data exists in text style transfer, seq2seq models are typically used Jin et al. (2022). Thus, for our parallel text fact transfer setting, we compare supervised ModQGA (ModQGA-Sup) with the following supervised seq2seq models:
1) $z$-Shot GPT: We construct a $z$-shot prompt for GPT-3.5 to generate the target text with the source text, source topic, and target topic as inputs. This model relies on its internal knowledge.
2) $z$-Shot GPT+Retr: We add the top-5 retrieved facts from $\mathcal{C}$ as an extra input to $z$-Shot GPT.
3) LED: LED Beltagy et al. (2020) is a seq2seq LM based on the Longformer. LED produces the target text using the source text, source topic, target topic, and corpus as inputs. This model is Mod-QGA-Sup without the transferred entities as inputs.

All GPT-3.5 models are gpt-3.5-turbo with a temperature of 0.2. Models that perform retrieval use the same Contriever setup Izacard et al. (2022) as ModQGA. The input query used is the source text $\mathcal{D}_{s}$ with every occurrence of $t_{s}$ replaced with $t_{t}$. We found that this query outperforms solely the target topic $t_{t}$, as it provides the Contriever context as to which information to search for (see Table 6).

We measure the output similarity of the predicted and target texts with ROUGE-1/2 (R1/R2) and BLEU Lin (2004); Papineni et al. (2002), serving as proxies for style preservation of the source text.

To evaluate factuality, we adopt three metrics: 1) Halluc calculates the average percentage of tokens that are extrinsically hallucinated, meaning that they do not appear in the corpus $\mathcal{C}$ or source text $\mathcal{D}_{s}$; 2) FactCC Kryscinski et al. (2020) is a classifier that predicts if any factual errors exist between a source text and claim. We use the true output as the source and each sentence of the generated text as the claim, and report the proportion of sentences with no factual errors; 3) NLI-Ent uses textual entailment to predict whether a claim is entailed by a source Maynez et al. (2020). We train a DistilBERT Sanh et al. (2019) classifier on the MNLI dataset Williams et al. (2018) (accuracy of 0.82) and report the proportion of sentences in the generated text that are entailed by the true output. All metrics are reported from a single run.

In Table 1, we see that 0-shot ModQGA excels at text fact transfer, achieving the strongest results in 22/24 metrics. This is impressive given that ModQGA has significantly less parameters than GPT-3.5 (0.8B vs 175B). We also note that 0-shot ModQGA outperforms ModQGA-Sup on open-domain t-REX, showing that our 0-shot model is more adaptable than its supervised version, but is surpassed by BART+Retr, opening the door to research in 0-shot text fact transfer to close this gap.

In Table 2, we find that ModQGA-Sup outperforms baselines on three datasets (17/18 metrics on U.S. News/Medline/Google), and achieves the second strongest results on t-REX. Further, ModQGA-Sup surpasses LED in 23/24 metrics, meaning that our extra input of transferred entities is valuable for improving the style and factuality of seq2seq models in text fact transfer. These findings suggest that our strategy of identifying entities, transferring entities between topics, and infilling, can outperform generating text solely in a seq2seq manner.

Finally, we note that GPT-3.5 fails to produce factual text, obtaining much lower factuality scores that are not always improved by using the corpus $\mathcal{C}$. The LLM also struggles to adhere to the style of the source text, shown by the lower output similarity scores and larger length ratios. Thus, text fact transfer highlights the limitations of GPT-3.5 with preserving factuality and style, meaning that our task could benchmark these capabilities of LLMs.

We invite two computer science and engineering students to evaluate 50 generated outputs from U.S. News and Google on style (i.e. which output best matches the source text style) and factuality (i.e. which output is more factual). Following best practices, we use a pairwise comparative evaluation Lewis et al. (2020b). To study the issues of 0-shot LLMs, we compare 0-shot ModQGA and 0-Shot GPT+Retr, and to study if the extra inputs of transferred entities aid seq2seq models, we compare ModQGA-Sup and LED.

In Table 3, the evaluator ratings indicate that 0-shot ModQGA better preserved style compared to 0-Shot GPT+Retr in over 55% of cases on both datasets and was more factual on U.S. News in 47% of cases, highlighting that ModQGA is a preferred choice for the challenging task of 0-shot text fact transfer. Further, evaluators indicated that ModQGA-Sup outperformed LED in factuality in 46% of cases on U.S. News, once again suggesting that our transferred entities can improve the factual accuracy of seq2seq models. These findings parallel our quantitative results (section 5.1), reinforcing that ModQGA can effectively transfer factual content without sacrificing the style of the source text.

We conduct an ablation study (Table 4, full results Appendix 8) and note that the use of generic questions and specificity guidance improve the output similarity and factuality of 0-shot ModQGA. We find the specificity result to be noteworthy, as it means controlling specificity can enhance the performance of 0-shot text fact transfer frameworks.

In Figure 3, we assess the abilities of our specificity-aware QA model. Overall, we find that the model does use the specificity of the entity guidance, having the ability to provide a regional descriptor (“residential”), city (“Tallahassee”), city and state (“Tallahassee, Florida”), and city descriptor (“The State Capital”). This suggests that our model has at least some ability to control the specificity of its answers.

Despite these strengths, our QA model may still err. Specifically, the model may identify a part of the context that matches the specificity of the entity, even though it does not correctly answer the question (e.g., “North” comes from the context “North side of campus,” but the correct answer is “South”). Further, the model may be biased towards answers that match the length of the entity, even if the specificity is not matched (e.g., predicting “Tallahassee” instead of “Florida”). Finally, if the provided entity is drastically unrelated to the question (e.g., “200”), so will the answer (e.g., “185”). Controlling specificity is a difficult task Huang et al. (2022), but we believe our specificity-aware QA model reveals a potential direction to address this problem.

In Appendix B.3, we present examples of target texts generated by ModQGA and other baselines.