FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Pre-trained language models are classified into three categories: autoregressive language models (such as GPT), masked language models (such as BERT), and encoder-decoder models (such as BART, T5).

This paper uses three tokens, Chinese Pre-trained language Model (CPM) [22], English Pre-trained language Model (EPM) [22], and Generative Pre-Training (GPT) [11], to represent autoregressive language models. We use BERT [4] to represent mask language models. Moreover, we use three tokens, Cost-efficient Pre-trained language Models (CPM-2) [21], cost-efficient English Pre-trained Language Models (EPM-2) [21] and Transformer [17], to represent encoder-decoder models.

Autoregressive language models predict the next word $x$, given all the previous words $x_{1}$, $x_{2}$, …, and $x_{i-1}$. The training goal is to maximize the log likelihood $\sum_{i}m_{i}\log\left(P\left(x_{i}\mid x_{1},\ldots,x_{i-1},x_{i+1},\ldots,x_{n}\right);\theta_{T}\right)$, where $\theta$ is the model parameter. Typical autoregressive language models include GPT, GPT-2 [12] and GPT-3 [2].

Mask language models use the special token [MASK] to randomly select the word to be masked, or replace it with a random token. This architecture forces the model to collect bidirectional information when making predictions. Popular representations of MLM include BERT [4] and RoBERTa [8]. Specifically, MLMs such as BERT use Transformer encoder architecture. Like the autoregressive model, MLM stacks multiple Transformer encoder layers to learn increasing complex and meaningful representations, but when learning the representation of a specific token, it uses masked self-attention to focus on all others in the sequence token.

Encoder-decoder models use more flexible “text input, text output” pattern, which learns to generate tokens $y_{1}$, …, $y_{n}$ with the given input sequence $x_{1}$, …, $x_{m}$. Representative models include BART [6] and T5 [13].

Autoregressive models include 2.6-billion-parameter CPM [22], 2.7-billion-parameter GPT-Neo, 6-billion-parameter GPT-J and 175-billion-parameter GPT-3 [2].

Mask language models include 330-million-parameter Roberta model, 27-billion-parameter PLUG [7] model, 110-billion-parameter Pangu [20] model, and 1.6-billion-parameter CTRL model.

Encoder-decoder models include 11-billion-parameter CPM-2 [21] model.

The Table-1 shows the popular models performance on GLUE-QQP test set. In the Section 4.3, we will compare the performance of ours with existing popular models.

In experiment of the original GPT article, Radford et al. use multi-layer transformer decoders in the language model, which is a variant of Transformer. It applies a multi-head self-attention operation to the input tokens, and a position feed-forward layer to generate the output distribution on target tokens.

We study the performance impact of the model depth by testing the CPM model with 36, 64, 128 layers and the EPM model with 36, 50, 64 and 80 layers. All other hyper-parameters are identical with the original models. We expanded network layers number from 32 layers to 36, 64, and 128 for the CPM model, and we have expanded layers from 32 to 36, 50, 64, and 80 for the EPM model.

BERT is a pre-trained Transformer network that sets up the latest state-of-the-art results for various natural language processing tasks, including question answering, sentence classification, and sentence pair regression.

The BERT-Large model mentioned in the paper by Devlin et al. consists of 24 self-attention layers. In this study, we extend the original network from 24 layers to 50, 60, 70, 80, and 90 layers, with all other hyper-parameters unchanged. The performance impact of the model depth is also discussed in Table-3.

CPM-2 is a standard Transformer model that combines a bidirectional encoder and a directional decoder. In order to reduce memory consumption and accelerate pre-training, Zhang et al. use mixed-precision training, gradient checkpointing, and zero stage optimization [14].

Most powerful neural sequence transduction models have an encoder-decoder structure. Transformer [17] follows this overall architecture, and uses stacked self-attention and point-by-point fully connected encoder and decoder layers.

We reduce the original CPM-2 model from 48 layers of the Transformer unit to 12 and 24 layers. All other hyper-parameters in our model are consistent with the CPM-2 model in the original paper. Moreover, we train the 12-layer EPM-2 model on English datasets.

In this work, we focus on three models: GPT-2, BERT, CPM-2, a language model based on a left-to-right generative transformer, a bidirectional transformer model based on masking, an encoder-decoder language model, respectively.

The infrastructure is optimized for multi-node deep learning applications. All experiments use up to 2 DGX servers (a total of 16 A100 SXM3 40GB GPUs). Through NVSwitch, we achieved a bandwidth of 300GB/sec between GPUs in servers and achieved a bandwidth of 10GB/sec between servers that use 1 InfiniBand adapter per server.

We describe the most significant models as follows:

• •

  Based on GPT-2, we trained a 10.3-billion-parameter model on Chinese datasets and we trained a 2.9-billion-parameter model on a dialogue corpus. We trained a BERT model with 495 million parameters on Chinese datasets. Moreover, we trained a Transformer model with 5.6 billion parameters on Chinese datasets.

• •

  We apply the corresponding training work for English. Using the GPT-2 model, we trained a model with 6.4 billion parameters on English datasets. We trained a BERT model with 1.24 billion parameters on English datasets and we trained a language model with a 688 million parameter on one GPU. We trained a Transformer model with 2.9 billion parameters on English datasets.

In evaluation stage, we evaluate five BERT-E models on the QQP classification task from GLUE and evaluate BERT-C model on the TNEWS classification task from CLUE.

From Table-2, this work has trained many Chinese and English corpora and conducted detailed training.

The models with the least number of layers are CPM-2-X and EPM-2-X, with only 12 layers. Model with the most significant number of layers is CPM-X, with 128 layers trained. From perspective of the difficulty of training, GPT structure is easier to cascade the number of layers. While Transformer structure is not easy to increase the number of layers, it is easier to reach the upper limit of hardware memory. From a perspective of model parameters, model with the least amount of parameters is BERT-C, with only 330 million, and the most significant parameter is CPM-X-L, with parameters reaching 10.3 billion.

For BERT, GPT, and Transformer, our network structure parameters used in this work are fixed, and we use parameters in the original paper without significant modifications.

We observe that in terms of time consumption, the shortest training time is CPM-X-S, and the longest time is CPM-X-EN from Table-4. The least training step number is CPM-X-S, only 100,000 steps. Moreover, the largest is CPM-X-EN with 2.8 million steps. From the perspective of computing power, the least computing resource is CPM-X-S, which uses 27 Eflops, and the most computing power is CPM-2-X-M, which uses 1,240 EFlops.

From Table-3, we observe that the best result does not come with the largest 90-layer-BERT-E-E model, the 70-layer-BERT-E-L model achieved the state-of-the-art result with about 0.5% gain to the 90-layer-BERT-E-E model. We can conclude that for the GLUE-QQP task, the optimal layer number for BERT models is 70. We conclude the result from the comparison with the current GPU hardware limits. In the future, the optimal layer number for the GLUE-QQP task might change with the help of advanced GPU servers.