MatInf: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization

In recent years, large-scale datasets (e.g., ImageNet Deng et al. (2009) and SQuAD Rajpurkar et al. (2016)) have inspired remarkable progress in many areas like Computer Vision (CV) and Natural Language Processing (NLP). On the one hand, well-annotated data provide essential information for training supervised machine learning models. On the other hand, benchmarked datasets make it possible to evaluate and compare the capability of different methods on the same stage.

Due to the high cost of data annotation, existing NLP datasets are usually labeled for only one particular task (e.g., SQuAD Rajpurkar et al. (2016) for question answering, CNN/DM Hermann et al. (2015) for summarization and AGNews Zhang et al. (2015) for text classification). These single-task datasets hinder the development of learning common and task-invariant knowledge Liu et al. (2017). Although multi-task learning and transfer learning have delivered encouraging results, we still cannot determine whether the improvement is from the extension of input or supervision. Furthermore, task-specific data make the models tend to learn task-specific leakage features Zhang et al. (2019) rather than meaningful knowledge that could generalize to other tasks. However, as a key step to Artificial General Intelligence (AGI), knowledge acquisition requires the model to learn more general knowledge instead of overfitting on a specific task. Therefore, a large-scale and cross-task dataset is in huge demand for future NLP research. Nevertheless, to the best of our knowledge, none of the existing datasets could meet such demand.

In this paper, we propose Maternal and Infant Dataset (MatInf), the first large-scale dataset covering three major NLP tasks: text classification, question answering and summarization. MatInf consists of question answering data crawled from a large Chinese maternity and baby caring QA site. On this site, users can ask questions related to maternity and baby caring. When submitting a question, a detailed description is required to provide essential information and the asker also needs to assign a category for this question from a pre-defined topic list. Each user could submit an answer to a question post, and the asker will select the best answer out of all the candidates. To attract more attention, the askers are encouraged to set rewards using virtual coins when submitting the question and these coins will be given to the user who submitted the best answer selected by the asker. This rewarding mechanism could constantly ensure high-quality answers.

MatInf supports three NLP tasks as follows. Text Classification. Given a question and its detailed description, the task is to select an appropriate category from the fine-grained category list. Different from previous news classification tasks whose category set is general topics like entertainment and sports, MatInf-C is a fine-grained classification under a single domain. That is, the distance between different categories is smaller, which provides a more challenging stage to test the continuously evolving state-of-the-art neural models.

MatInf holds the following merits: (1) Large. MatInf includes 1.07M unique QA pairs, making it an ideal playground for the new advancements of deeper and larger models (e.g., Pretrained Language Models). (2) Multi-task applicable. MatInf is the first dataset that simultaneously contains ground truths for three major NLP tasks, which could facilitate new multi-task learning methods for these tasks. Here, to set a baseline and inspire future research, we present Multi-task Field-shared Sequence to Sequence (MTF-S2S), a straightforward yet effective model, which achieves better performance on all three tasks compared to its single-task counterparts.

We present Maternal and Infant (MatInf) Dataset, a large-scale dataset jointly labeled for classification, question answering and summarization in the domain of maternity and baby caring in Chinese. An entry in the dataset includes four fields: question (Q), description (D), class (C) and answer (A). An example is shown in Figure 1, and the average character and word numbers of each field are reported in Table 1.

We collect nearly two million question-answer pairs with fine-grained human-labeled classes from a large Chinese maternity and baby caring QA site. We conduct both automatic and manual data cleansing and remove: (1) classes with insufficient samples; (2) entries in which the length of the description filed is less than the length of the question field; (3) data with any field longer than $256$ characters; (4) human-spotted ill-formed data. After the data cleansing, we construct MatInf with the remaining $1.07$ million entries.

We first randomly split the whole data into training, validation and test sets with a proportion of 7:1:2. Then, we use the splits for summarization and QA. For classification, we further divide the data into two sub-tasks according to different classification standards within each split.

Recently, many attempts have been made on multi-task learning in NLP Liu et al. (2015); Luong et al. (2016); Guo et al. (2018); McCann et al. (2018); Xu et al. (2019); Ruder et al. (2019); Liu et al. (2019); Radford et al. (2019); Dong et al. (2019); Shen et al. (2019); Raffel et al. (2019); Lei et al. (2020) and several benchmarks are available for multi-task evaluation Wang et al. (2019a, b). Though recent studies show that multi-task learning is effective, there is still one more question to answer. That is, when training models on multiple tasks, multiple datasets are used by default. As illustrated in Figure 2(a), it adds both new input (i.e., text, denoted as $X$) and new supervision (i.e., ground truths, denoted as $Y$). Due to the different processes of data collection, $X$ in different datasets have different sources and properties. Recent progress on Language Modeling Radford et al. (2019); Devlin et al. (2019); Yang et al. (2019); Raffel et al. (2019) has proved that corpora ($X$) from different sources can make the model more robust and significantly improve the performance. To this end, it is not easy to determine whether the success of a multi-task model should be mainly attributed to the addition of $X$ or $Y$. However, as depicted in Figure 2(b), in MatInf, our jointly labeled fashion can guarantee that $X$ remains the same as in a single task and only $Y$ is added. Thus, MatInf provides a fair and ideal stage for exploring multi-task learning, especially auxiliary and multi-task supervision under a single dataset.

To set a baseline and also inspire future research, we design a multi-task learning network, named Multi-task Field-shared Sequence to Sequence (MTF-S2S). We illustrate the architecture of MTF-S2S in Figure 3. For generation tasks, we combine the summarization ($D\rightarrow Q$) and QA ($Q\rightarrow A$) to be the form of $D\rightarrow Q\rightarrow A$, with a shared Long Short-Term Memory (LSTM) for decoding questions in summarization task and encoding questions for both QA and classification tasks. Previous studies often share layers among tasks to regularize the representation learning, as illustrated in Figure 2(c). Different from that, MTF-S2S shares on both module level (i.e., field encoder/decoder, as shown in Figure 2(d)) and layer level. An attention mechanism is applied when decoding for summarization and QA. Also, we concatenate the encoded representations of description and question, and feed it to a shared fully connected layer and then specialized fully connected layers for age classification and topic classification, respectively.

When training, since the sizes of datasets for different tasks are not equal, we first determine the batch size for different tasks to make sure that the training progress for each task is approximately synchronized by:

$$\forall a,b\in T,bs_{a}/bs_{b}=n_{a}/n_{b}$$

(1)

where $T$ includes four tasks: summarization, QA, and two classification tasks. $bs_{*}$ is the batch size of each task, and $n_{*}$ is the sample numbers in each dataset for the task. If one task is iterated to the last data batch, it will start over from the first batch. For each iteration, we successively calculate the losses by Cross Entropy for each task in one batch. Then, we train the model to minimize the total loss:

$$\mathcal{L}=\sum_{t_{i}\in T}\lambda_{i}\mathcal{L}_{i}$$

(2)

where $\lambda_{*}$ is the manually set weight for each task. We stop the co-training after one epoch, then fine-tune the model to obtain the peak performance for each task, separately.

In this section, we benchmark a few baselines and MTF-S2S on the three tasks of MatInf. We run each experiment with three different random seeds and report the average result of the three runs.

Since MatInf is a web-crawled dataset, it would be inevitable to be noisier than a dataset annotated by hired annotators though we have made every effort to clean the data. On the bright side, it can encourage more robust models and facilitate real-world applications. For future work, we would like to see more interesting work exploring new multi-task learning approaches.

To conclude, in this paper, we present MatInf, a jointly labeled large-scale dataset for classification, question answering and summarization. We benchmark existing methods and a straightforward baseline with a novel multi-task paradigm on MatInf and analyze their performance on these three tasks. Our extensive experiments reveal the potential of the proposed dataset for accelerating the innovations in the three tasks and multi-task learning.

We are grateful for the insightful comments from the anonymous reviewers. This research was supported by National Natural Science Foundation of China (No. 61872278). Chenliang Li is the corresponding author.