Overview of the VLSP 2022 - Abmusu Shared Task: A Data Challenge for Vietnamese Abstractive Multi-document Summarization

The data is automatically collected and filtered from Vietnamese electronic news on $8$ categories, including the economy, society, culture, science and technology, etc. It is divided into training, validation and test datasets. The datasets contain several document clusters. Each cluster has $3-5$ documents that illustrate the same topic. On training and validation datasets, a manual-created reference abstractive summary is provided per cluster. The test set is formatted similarly to the training and validation sets, but without an abstractive summary.

The data preparation process is described in Figure 1. We used INCEpTION³³3http://https://inception-project.github.io// Klie et al. (2018) as the annotation tool. It is a semantic annotation platform offering intelligent assistance and knowledge management. There are $10$ human annotators and $2$ experts who participated in the annotation process, the annotation guideline with full definition and illustrative examples was provided. We used an $8-$step process to make summarization data, each data sample needs the involvement of at least $1$ annotator and $1$ reviewer:

• •

  Crawl data from news websites by categories.

• •

  Group documents into clusters by the highlighted hashtag, category, posted time, and similarity.

• •

  Remove duplicate or highly similar documents.

• •

  Remove clusters with too few articles, and review to select clusters/documents manually.

• •

  Choose $200$ more clusters randomly to ensure the distribution for difficult test-cases.

• •

  Create the summary manually by the annotators.

• •

  Re-check the quality of the summary (by the reviewers) to ensure the quality and length. Unqualified data is relabeled by another annotator.

• •

  Refine all data by expert reviewers.

As a result, we prepared a total of $1,839$ documents in $600$ clusters: $621$ documents ($200$ clusters) for the training set, $304$ documents ($100$ clusters) for the validation set and $914$ documents ($300$ clusters) in the test set. Figure 2 show the distribution of categories in the training/validation set and the test set. Table 1 and Table 2 describe the statistics of the Abmusu dataset in detail at the token- and the sentence level. The compression ratio of Abmusu dataset is $\sim 9\%$, the manually created summaries often contain $4-6$ sentences.

Each data example includes the title, anchor text and body text of all single documents in a cluster. Each cluster also has a category tag and a manually created summary. The provided test set for the participated team is formatted similarly to the training and validation data, but without the manually created summary. The evaluation was performed on the AIhub⁴⁴4http://aihub.ml/ platform for $7$ days. Test data was divided into two parts: Public Test and Private Test, each containing $50\%$ of the test data. The Private Test was opened $4$ days after the Public Test. Each team is allowed to submit a maximum of $35$ submissions to the Public test ($5$ per day) and $5$ submissions to the Private Test (not limited per day).

The official evaluation measures are the ROUGE-2 scores and ROUGE-2 F1 (R2-F1) is the main score for ranking. ROUGE-2 Recall (R2-R), Precision (R2-P) and R2-F1 between predicted summary and reference summary are calculated as Lin (2004):

The committee provided $4$ baselines as the shared task benchmark, includes:

• •

  Ad-hoc rule-based baseline: The summary is the concatenation of the first and the last sentences of all component documents in each cluster.

• •

  Anchor text-based baseline: The summary is the concatenation of the anchor text of all component documents in each cluster.

• •

  Extractive baseline: The summary is generated by the extractive summarization model using Lexrank Erkan and Radev (2004) and MMR Goldstein and Carbonell (1998).

• •

  Abstractive baseline: The summary is generated by the abstractive summarization model ViT5 Phan et al. (2022).

There are $46$ registered teams from research groups in domestic and international Universities (VNU-HUS, VNU-UET, HUST, PTIT, etc.) and industries (Viettel, VinGroup, CMC, TopCV, VCCorp, etc). In which, 28 teams submitted the data agreement, and 16 teams participated officially by submitting at least 1 run on the evaluation platform. Participant teams can use all possible tools and resources to build models. Participated teams made a total of 287 submissions. Post-challenge panels⁵⁵5http://aihub.ml/competitions/341 are now opened on AIHUB for supporting research improvements.

An interesting observation is that the rule-based baseline achieved surprisingly high results (ranked 6). This result can be explained because most news are written in an explanatory or inductive style, so the first and last sentences often contain important information. The extractive baseline result (ranked 5) was much better than the anchor text baseline result (ranked 18), contrary to the assumption that the anchor text can be considered as a simple summary of the news text. In the abstractive baseline model, we only put raw data through the ViT5 model without any parameter tuning, so it is reasonable when its result was low (ranked 19).

The proposed models followed two main approaches: abstractive summarization and hybrid approach. Participated teams used a variety of techniques, including similarity scoring (TF-IDF, Cosine, etc,), graph-based methods (i.e., Lexrank Erkan and Radev (2004), Textrank Mihalcea and Tarau (2004), Pagerank Brin and Page (1998), etc.), sentence classification (Long short-term memory Hochreiter and Schmidhuber (1997), BERT Kenton and Toutanova (2019), etc.) and text correlation. The results of the private test were considered as the official results to rank the team in Abmusu shared task. The results on ROUGE-2 of the top 5 teams and 4 baselines are shown on Table 3 (See Appendix A for the full results.). All $16$ teams achieved performance higher than the anchor text baseline and abstractive baseline. There were $5$ teams that achieved a higher F-score than our extractive and rule-based baselines. The best ROUGE-2 F obtained was $0.3035$, the corresponding ROUGE-2 P and ROUGE-2 R are $0.3035$ and $0.2298$ respectively.