PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency

Initially, we constructed the Japanese corpus from CommonCrawl using CCNet. However, for the latter portion consisting of 0.5 trillion tokens, we undertook a comprehensive preprocessing effort from the ground up, using 20 data dumps spanning from 2017 to 2024, thereby generating a dataset comprising approximately 460 billion tokens.

There are two main reasons for this approach:

• •

  The WET files processed by CCNet lack the structured information that is typically available in the HTML or Markdown format; consequently, we opted to process the WARC files directly.

• •

  Accumulated expertise during this project has bolstered our confidence that we can efficiently generate datasets independently.

The processing pipeline, akin to RefinedWeb (Penedo et al., 2023) and the Swallow Corpus⁴⁴4https://tokyotech-llm.github.io/swallow-corpus, managed the raw archived data stored in the WARC format through the following steps:

1. 1.

   Download WARC files while extracting Japanese HTML and text files.

2. 2.

   Convert the extracted data to Markdown if it is in HTML format.

3. 3.

   Filter the entire document using llm-jp corpus filter.

4. 4.

   Conduct deduplication across all dumps employing the MinHash algorithm.

5. 5.

   Re-shard the data into uniformly sized chunks, suitable for dataset consumption.

CommonCrawl comprises collections of segmented files ranging in size from several GiB, thereby facilitating embarrassingly parallel processing with the exception of the MinHash step. For this task, we executed each process in parallel using 1 000 instances. Steps 1 through 3 required approximately 30 hours per dump, step 4 took about 24 hours, and step 5 was completed within several hours.

The architecture of PLaMo-100B closely resembles that of Llama2⁵⁵5https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/ and Llama3⁶⁶6https://ai.meta.com/blog/meta-llama-3/. To enhance training stability, we implemented QK Normalization (Wortsman et al., 2024) because Wortsman et al. (2024) indicate that QK Normalization effectively stabilizes computations within self-attention layers and contributes to the overall stability of model training.

In preliminary experiments, we verified that QK Normalization does not adversely affect model performance. Additionally, findings in other studies, such as Jamba (Lieber et al., 2024) and Chameleon (Team, 2024), demonstrate that incorporating a normalization layer prior to the interaction between tokens enhances training stability. This observation suggests that QK Normalization may emerge as a standard technique for large-scale models.

Regarding the loss function, we incorporated z-loss, which enhances numerical stability within the softmax cross-entropy loss function, defined as

$$L(x)=\left(\log\left(\sum_{i=0}^{C}\exp\left(x[i]\right)\right)\right)^{2},$$

(1)

where $x\in\mathbb{R}^{C}$ is the output of PLaMo-100B for the next token prediction and $C$ is the vocabulary size. Similar to QK Normalization, Wortsman et al. (2024) demonstrated that z-loss contributes to the stabilization of training processes.

In preliminary experiments, we verified that z-loss does not adversely affect model performance, akin to QK Normalization. While it remains uncertain whether z-loss achieves its intended purpose of stabilizing training, we have yet to observe any negative impact from its implementation. Furthermore, z-loss has proven to be a valuable metric for monitoring training progress. In instances where training deviates from expectations (such as due to a bug), the changes in z-loss are often more significant than those observed in other loss functions or in downstream task performance. This characteristic facilitates the identification of whether observed changes are attributable to trial-to-trial variability or other factors.

While we have implemented two methods that contributed to the stabilization of training, we also explored additional techniques. In this section, we discuss those methods that either exhibited no discernible effect or had a counterproductive impact on training stability.

For pre-training, we implemented 3D parallelism (Shoeybi et al., 2020), a method that integrates data parallelism with two types of model parallelism: tensor parallelism and pipeline parallelism to enable the training of large-scale models.

As for pipeline parallelism, we adopted Zero Bubble (Qi et al., 2024). It is recognized that pipeline parallelism may encounter inefficiencies due to periods in which certain GPUs remain idle, known as “bubbles.” However, Zero Bubble aims to effectively minimize these idle periods to zero.

We did not implement the speculative parameter updates introduced by Zero Bubble for the following reasons:

• •

  The definition of one iteration becomes ambiguous in the Python script, complicating the debugging process.

• •

  Gradient clipping was consistently applied, resulting in very few iterations where speculative execution could be deemed effective.

The second point stands in contrast to the findings reported in the original paper, which suggested that gradient clipping is infrequently applied. We posit that this discrepancy may stem from differences in model size between our setup and that used in the study.

LLMs, including PLaMo-100B-Base, are composed of repeated Transformer blocks but necessitate a final linear layer to predict the next token. In models provided via Hugging Face’s Transformers library (Wolf et al., 2020), this layer is commonly referred to as the lm-head.

Initially, we computed this layer in FP8 format to enhance training speed. Although we observed minimal issues concerning training loss, we experienced suboptimal performance in subsequent benchmark tasks. Through investigations conducted in smaller experimental settings, we discovered that the z-loss values were significantly elevated when using FP8 for the lm-head as shown by Figure 2, suggesting that the lm-head should be computed in a higher precision format than FP8 format. In our case, we used bfloat16 for the lm-head to mitigate this issue.

The pretraining of PLaMo-100B-Base achieved a computational speed of approximately 540 TFLOP/s/GPU, which is about 27% of the theoretical speed of 1 979 TFLOP/s for FP8 on the H100. While a direct comparison is difficult due to differences in the number of GPUs used, we believe that our performance is comparable to that of Llama3 and the benchmarking by MosaicML⁷⁷7https://github.com/mosaicml/llm-foundry/tree/main/scripts/train/benchmarking.

In Supervised Fine-Tuning (SFT), the training process is guided by a predefined dataset comprising paired examples of “questions posed to the LLM” and their respective “expected answers from the LLM”. This dataset encompasses a variety of tasks, each consisting of a question-answer pair. For instance, in the domain of mathematical problem-solving, each pair entails a problem and its respective solution. Similarly, in conversational settings, each pair consists of a user prompt and the preferred response. The objective of using a training dataset of high diversity and quality is to enhance the downstream task performance of the LLM.

Training in SFT is conducted using next-token prediction, analogous to the pre-training phase, albeit with an emphasis on fine-tuning the response generation component. To optimize this process, only the response portion of the input text is considered during loss calculation rather than the entire input sequence. Although our preliminary experimental evaluations revealed no significant performance difference between this approach and that considers the entire input, this method was adopted to mitigate the risk of the model internalizing undesirable traits, such as aggressive expressions, potentially present in the questions.

Traditionally, in SFT, various tasks’ training datasets are aggregated and trained simultaneously. However, some prior studies, such as Nemotron-4 (Adler et al., 2024), have documented conflicts arising from concurrent learning of multiple tasks. Despite attempts to alleviate these conflicts through adjustments to the sample weighting ratios, attaining a substantial level of harmony was challenging, especially in coding tasks. Consequently, a two-stage SFT approach is proposed wherein the coding tasks are trained in the first stage of SFT, followed by the second stage of SFT, which focuses on more general tasks.

In our experiment, a similar trend was observed for mathematical questions. Therefore, we adopted the two-stage SFT method, initially segregating mathematical questions for the first stage and subsequently addressing various other tasks in the second stage.

Direct Preference Optimization (DPO) is an algorithm proposed by Rafailov et al. (2023) that learns human preferences from labeled data, wherein pairs of responses to the same question are designated as either better (chosen) or worse (rejected). The model is encouraged to generate more preferable responses by leveraging this preference information.

In the learning process of DPO, several existing datasets, such as hh-rlhf (Bai et al., 2022) and HelpSteer2 (Wang et al., 2024), which include labeled responses generated by LLMs or human authors, are commonly used. Alternatively, data can also be synthesized by allowing the model to generate multiple response candidates, which can then be labeled accordingly. The former scenario, where the model producing the responses differs from the one being trained, is referred to as off-policy, while the latter scenario, in which the model being trained generates the responses, is termed on-policy.

On-policy training has demonstrated effectiveness in enhancing learning efficiency, as it provides preference feedback on the types of responses the model is more likely to generate (Tajwar et al., 2024). A hybrid approach, known as SPIN (Chen et al., 2024), has also been introduced, which involves generating a dataset by pairing teacher responses from the SFT dataset with model-generated responses, under the assumption that the teacher’s responses are more preferred.

When employing on-policy data for training, it is possible to alternate between data generation and DPO training termed Iterative DPO. This method has been shown to yield superior results compared to a single round of DPO (Xu et al., 2024; Dong et al., 2024).

Three different datasets are combined for our two-stage DPO training after SFT: (1) a publicly available dataset, (2) a dataset generated by labeling responses produced from a snapshot of PLaMo-100B, and (3) a dataset generated through the SPIN methodology. This approach enables us to take advantage of both high-quality publicly available preference datasets and the efficacy of Iterative DPO using on-policy datasets. The details of the data generation process are described in the data generation section.

Model merging is a technique that integrates multiple models to enhance overall performance (Izmailov et al., 2019; Wortsman et al., 2022). Various methodologies exist for model merging; for instance, Llama-3.1 (Llama Team, AI @ Meta, 2024) has reported using the average of multiple models. We employed a straightforward model merging technique known as Spherical Linear Interpolation (SLERP) (Shoemake, 1985), which computes the midpoint between two models.

There were several DPO training results, depending on the combination of training data and other factors such as hyperparameters. By merging two of these models with distinct characteristics, we were able to create a model that incorporates the strengths of both models to some extent.

High-quality post-training datasets suitable for commercial use are available, such as oasst2 (Köpf et al., 2023) or hh-rlhf (Bai et al., 2022) in English, along with ichikara-instruction⁸⁸8https://huggingface.co/datasets/p1atdev/ichikara-instruction in Japanese. Furthermore, the quantity of publicly accessible datasets is continuously increasing. We conducted experiments on a variety of these datasets to decide our instruction-tuning dataset collections.

To accurately address mathematical problems, we developed templates for various problem types that require calculations, subsequently generating datasets by varying the numerical values. Our mathematics dataset was manually constructed without machine learning techniques. Although there is a limit to the number of problem templates that can be created manually, and many data points would have only different numerical values, we decided it would be okay based on our previous empirical studies and the considerations outlined below.

When an LLM produces a calculation result, the distribution of tokens tends to exhibit a deterministic quality, which aligns with our objectives during the training process. Furthermore, the textual content outside of the mathematical formulas is likely to adhere to a standardized format. Even when the only difference between data points is the numerical values, some degree of diversity in outcomes may still be observed. This includes distinctions such as whether carrying occurs during addition, the potential for simplification of fractional results, or the choice of which variable to eliminate in simultaneous linear equations.

Existing datasets that do not rely on machine learning include the AMPS pretraining corpus (Hendrycks et al., 2021) and the work by Saxton et al. (2019). However, these datasets feature artificial TeX representations for formulas, and their answers are restricted to numerical values, indicating potential areas for enhancement in post-training applications. To address this, we generated our own mathematical datasets aimed at augmenting the volume of Japanese mathematical data. We have used the math-related datasets for pre-training as well, but for post-training, we apply a different format, such as instruction-based responses, and combine different datasets with varying ratios, taking into account the characteristics of each dataset.

For the question-answering dataset, we employed the Self-Instruct algorithm (Wang et al., 2023) as a foundation for data generation. However, rather than using the algorithm directly with GPT, we developed a method to facilitate the data generation using smaller LLMs like PLaMo-13B (Preferred Networks, 2023). For instance, when attempting to generate a question sentence directly, the results were suboptimal, prompting us to incorporate an additional step to first generate a concise title.

During the development of PLaMo-100B, we also focused on translating the collected and generated datasets into Japanese. The availability of post-training datasets for commercial use in Japanese is severely limited, creating challenges in acquiring a sufficient quantity and diversity of training data. Even when generating our own data, numerous open LLMs are primarily designed in English, complicating the generation of high-quality Japanese responses. By using PLaMo-100B itself for translation, we successfully increased the volume of high-quality Japanese data, resulting in performance enhancements in Japanese text generation tasks.

We generated preference data during the post-training process of PLaMo-100B. Referring to the work by Dong et al. (2024), we generated eight different responses for the same prompt using PLaMo-100B and evaluated their scores. The highest-scoring response was selected as the “chosen” response, while the lowest-scoring one was marked as “rejected”. To evaluate response scores, we experimented with both the LLM-as-a-Judge method using open LLMs (Zheng et al., 2023; Verga et al., 2024) and the reward model.

In this data generation process, only the prompt is required, and a teacher response example is unnecessary. We can can datasets like chatbot_arena_conversations (Zheng et al., 2023), which only contains user prompts in a commercially usable license. During the response generation where LLM inference is required, we used vLLM (Kwon et al., 2023) for acceleration.