Can Language Models Make Fun? A Case Study in Chinese Comical Crosstalk

Depending on the number of performers, crosstalk is typically performed as a dialogue between two performers called ‘对口’, or rarely as a monologue by a solo performer called ‘单口’ (like stand-up comedy in the Western), or even less frequently, as a group acting by more performers called ‘群口’.

Let us take the dual performing (‘对口’) as an example. Dual performing usually involves two roles called Penggen ‘捧哏’ (Peng in short) and Dougen (‘逗哏’) (Dou in short). Dou aims to perform in a comical way using talking and actions. Peng is the support role to make the conversation more fluent and legible (As shown in Table 1). The conversation consists of a iterative sequence of utterances:

$$\Phi=\{{u}_{1},{v}_{1},{u}_{2},{v}_{2},\cdots,{u}_{K},{v}_{K}\}$$

which is a $K$-turn dual crosstalk conversation with 2K utterances including K utterances from Dou (denoted as $u$) and K utterances from Peng (denoted as $v$). Note that both $u_{i}$ and $v_{i}$ are utterances that consists of many utterances, namely $u_{i}=\{\phi_{i,1},\phi_{i,2},\cdots,\phi_{i,j},\cdots\phi_{i,l_{i}}\}$, $\phi_{i,j}$ is the $j$-word in the $i$-th Dou/Peng utterence and $l_{i}$ is the number of words in the utterence.

Training could be formulated as two paradigms: 1) a Seq2seq utterance generation task: it could be treated as a seq-to-seq task to predict the next utterance given previous utterances; 2) a next word generation task: it can also consider as a typical language model that does not consider the utterance border, namely a raw language model that predicts the next word. For automatic evaluation in Sec. 5, we adopts commonly-used generation metrics to evaluate models using an auto-regressive utterance generation manner, namely, predicting the next utterance based on previous utterances no matter it is trained in a Seq2seq utterance generation paradigm or next word prediction paradigm.

Crosstalk scripts (except for solo performers) are usually multiple-turn dialogues. It typically involves two (or more) performers talking about a topic in multiple turns (with an average of 72 in $C^{3}$ dataset), typically ranging from 10 to 20 minutes. In contrast to general dialogues, the characteristics of the crosstalk are as follows: 1) it is humor-oriented： it aims to make audiences laugh by freely talking. 2) it is with a novel language style: the crosstalk language itself is in a rapid, bantering, and highly interactive style. More interestingly, it is rich in puns and allusions. 3) it is culturally-grounded: it typically relates to not only the local daily life (especially in the north of China, e.g., Beijing) but also the long historical events in china with a time range from 3000 BC to the present. Interestingly, it usually adopts the Beijing dialect (closed to Mandarin) during some periods. 4) it is low-resourced: crosstalk generation task could rely on relatively low-resourced digitized scripts.

We collect data from the book ’ Encyclopedia of Chinese Traditional Crosstalk’ and the internet. The creation date of these scripts ranges from Qing Dynasty (roughly 1800) to this century. The main resources are from: 1) a digitized book named ‘ Encyclopedia of Chinese Traditional Crosstalk’ or 《中国传统相声大全》 published in 2003, which is a collection of traditional crosstalk collections, records, and compilations since Qing Dynasty; 2) many websites that maintain crosstalk scripts. See App B for more details. Our dataset uses the Apache-2.0 license.

We implement LSTM Seq2seq which is trained from scratch as a baseline. To make use of existing pre-trained language models, we also include pre-trained UniLM, GPT, T5 in a fine-tuned manner. Large-scale Chinese pre-trained language models like CPM, Zhouwenwang, Pangu-$\alpha$ were recently released, we therefore evaluate these models in a zero-shot fashion since fine-tuning on these models are economically-expensive. Furthermore, we also verified the effectiveness of GPT-3. Fortunately, GPT-3 provides an API for fine-tuning, making GPT-3 the only large-scale PLM that could be fine-tuned at an affordable cost.

LSTM Seq2seq Sutskever et al. (2014): The LSTM network consists of a two-layer bi-directional LSTM encoder and a two-layer LSTM decoder ⁵⁵5The codebase is from https://github.com/IBM/pytorch-Seq2seq. Both the embedding size and the hidden state size of the LSTM model are set to 300. The encoder-decoder model is augmented with an attention mechanism. For the k-th utterance in a dialog, the input of the encoder was the concatenation of all the past utterances before k truncated with 256 tokens, while the target output of the decoder was the k-th utterance.

UniLM Dong et al. (2019): Unified Language Model (UniLM) adopts multi-layer Transformers, which also uses different masks to control the number of visible context words thereby can be applied to both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Our pre-trained model is downloaded from ⁶⁶6https://github.com/YunwenTechnology/UniLM, pre-training with Wikipedia data and news corpus data in CLUE. The UniLM used in this paper consists of 12 layers with a hidden size of 768 and 12 heads. The ways to build fine-tuned data structures are the same as Seq2seq.

T5 Raffel et al. (2019) is a unified framework that treats various text tasks into a text-to-text format. It consists of an encoder component and a decoder component, both of which are a stack of many Transformer layers. We use the Chinese version of the T5 pre-training model, and use both T5-Chinese-base⁷⁷7https://huggingface.co/imxly/t5-pegasus and T5-Chinese-small⁸⁸8https://huggingface.co/imxly/t5-pegasus-small models for training. The parameters of the base model are 275 million, and the parameters of the small model are 95 million.

GPT Radford et al. (2018): Generative Pre-trained Transformer (GPT) models by OpenAI have taken the natural language processing community by introducing very powerful language models. The GPT model is based on a unidirectional transformer with some modifications. In our implementation, the GPT model is 12 layer Transformers with hidden size 768, pre-trained using LCCC Corpus Base corpus ⁹⁹9https://huggingface.co/thu-coai/CDial-GPT_LCCC-base and fine-tuned by crosstalk dataset. Follow the implement of code ¹⁰¹⁰10https://github.com/yangjianxin1/GPT2-chitchat, We divide the dialog into utterances, and sequentially combine utterances with fewer than 256 words as one input.

GPT-3 Brown et al. (2020): is a unidirectional language model, the biggest GPT-3 model uses 45TB of data for training and 175 billion parameters. At the same time, GPT-3 mainly focuses on the more general NLP model to solve problems with fewer domain data and no fine-tuning steps. Note that GPT-3 is mainly for English language generation, but it could also generate fluent Chinese texts. We applied the GPT-3 online test API ¹¹¹¹11https://beta.openai.com/ and evaluate crosstalk generation. GPT3-Davinci is the one with Davinci engine without fine-tuning. The real scale of Davinci engine is unknown since no details are exposed; however, some evidence suggests that Davinci engine might be the biggest model that is with 175B parameters ¹²¹²12https://blog.eleuther.ai/gpt3-model-sizes/. GPT3-Davinci-finetuned is the fine-tuned version using GPT-3 API. We fine-tune it on 200 crosstalk scripts in 4 epochs.

Pangu-$\alpha$ Zeng et al. (2021) is large-scale autoregressive language models, with up to 200 billion parameters. It consumes 1.1TB high-quality Chinese data from a wide range of domains. A publicly-available version of Pangu-$\alpha$ (with 2.6B parameters) could be used in https://huggingface.co/imone/pangu_2_6B.

CPM Zhang et al. (2021) is a generative pre-training model trained on 100 GB Chinese corpora. CPM-Large is with 36 transformer layer and reaches 2.6B parameters.

Zhouwenwang considers both the generative language model task and mask language model; it could have the ability for both language generation and natural language understanding. The larger model (Zhouwenwang-1.3B) is with 1.3 billion parameters ¹³¹³13https://github.com/IDEA-CCNL/Fengshenbang-LM.

We show an example of generated crosstalk scripts as shown in Table 6. All generated examples could be seen in https://github.com/anonNo2/crosstalk-generation.

Meaningless generation in LSTM Seq2seq LSTM language model produces fluent but nearly meaningless texts (annotated in gray color), this is probably due to the fact that the training data for Seq2seq models is not enough and no pre-training was adopted. While other models with pre-training do not frequently generate such nonsense texts. This shows that pre-training could boost the generation performance, especially for the scenarios with low-resourced training texts.

Repeated context topic in generation UniLM and GPT-3 could generate topic-coherent texts, especially, some generated utterances also repeat some key topics from the first 10 input utterances, e.g., ‘臭流氓’ (shameless rogues), ‘穿裤衩，到处跑’(running around in shorts), and ‘海尔兄弟’(Haier brother ¹⁵¹⁵15Haier Brothers, see https://www.imdb.com/title/tt8302572/, a cartoon about a pair of robots called ‘Haier Brothers’ who travel around the world to explore the nature.). Note in this example, the raw script (the last 10 utterances) dot not have so many repeated topics from previous utterances, like generation models.

Insulting words UniLM, GPT, and GPT-3 generate some insulting words that already appeared in the first 10 utterances, namely, ‘臭流氓’ (shameless rogues). Moreover, GPT also generates new insulting words, 就他长得这么丑 he just looks ugly that did not appear before. This is probably due to that the other training scripts or pretraining corpora may have such insulting texts.

Humorous generation Roughly, we could see some humorous generated utterances. For example, the last generated utterance for GPT-3 (in the last row and second last column) does have a sense of humor. However, if we treat these utterances as a whole, their performance of humor is not satisfied.

The difference between Peng and Dou Basically, Dou usually talks more and introduces more topics in dialogues while Peng usually supports Dou to make each of his topics more comprehensively talked and topic transfer more coherent. This leads to that Peng’s utterances sometimes contain only a few interjections like ‘嗯’(hum) and ‘哎哟’(ouch). We argue that the generation for Dou’s utterance is much more difficult than Peng, and the former is more interesting and worthy of more attention.