ABCI 3.0: Evolution of the leading AI infrastructure in Japan

AI Bridging Cloud Infrastructure (ABCI) is the world’s first large-scale open AI infrastructure, designed and developed by AIST with the aim of accelerating the development of AI technologies in Japan [1, 2, 3]. It has been installed in the AI Data Center located at AIST’s Kashiwa Center, and began operating in August 2018. To date, many organizations have achieved remarkable results by using ABCI, including the successful construction of Japanese large language models (LLMs) such as PLaMo [4], Swallow LLM [5], and so on. On the other hand, as the demand for generative AI in industry, academia, and government in Japan is growing rapidly, ABCI cannot provide enough resources to meet such user demands in a timely manner. In addition, the development of generative AI is still at an early stage and currently focuses mainly on natural languages. In the near future, it is important to develop real world multimodal foundation models that are built using large amounts of image, audio and sensor data obtained from the real world, such as manufacturing, transportation, and so on. In order to demonstrate such cutting-edge AI technologies, it is imperative to improve the computational capability of ABCI. Since November 2024, AIST has been gradually starting to operate the latest system in the ABCI series, ABCI 3.0, while leveraging the existing technical assets of ABCI. ABCI 3.0 will be fully operational in January 2025.

This paper is organized as follows. Section 2 introduces the hardware configuration of ABCI 3.0. In Section 3, we describe the initial software installed. The data center facility where we have upgraded for ABCI 3.0 is presented in Section 4. Finally, Section 5 briefly mentions the future perspective.

Figure 1 shows the overview of ABCI 3.0 hardware configuration. Table 1 shows the specifications for the ABCI series, from AAIC, which is the prototype system of ABCI, to ABCI 3.0. ABCI 3.0 consists of; 766 Compute Nodes (H) that form in total 6128 NVIDIA H200 GPU, shared file systems and cloud storage that provide in total 75PB capacity, InfiniBand NDR/HDR networks connecting the compute nodes and the storage systems, firewall equipments, and etc. ABCI 3.0 connects to the Internet through SINET6, the Science Information NETwork, at 400Gbps. Its peak performance is 6.22 exaflops in half precision and 3.0 exaflops in single precision, which is 7 to 13 times faster than the previous system, ABCI 2.0. It also more than doubles both storage capacity and theoretical read/write performance.

The compute node consists of HPE Cray XD670, that is equipped with two Intel Xeon Platinum 8558 Processors (2.1GHz, 48core) and eight NVIDIA H200 SXM GPUs, 2048 GiB DDR5-5600 memory, two 7.68TB U.3 NVMe SSDs, eight InfiniBand NDR200 adapters, and an InfiniBand HDR adapter.

The interconnection network adopts a three-tier fat-tree topology, and all compute nodes are connected with full bisection bandwidth. Each compute node has eight InfiniBand NDR200 HBAs, and the total injection bandwidth is 12 times larger than that of ABCI 2.0. Compute nodes and the storage system are connected via InfiniBand HDR network.

ABCI 3.0 achieves a highly cost-effective all-flash storage system by adopting QLC (Quad-Level Cell, 4-bit MLC) SSDs. The storage system provides Lustre parallel file system and AWS S3 compatible object storage.

ABCI 3.0 also has Interactive Nodes and Gateway Nodes. Interactive nodes are servers to login and used to compile programs and submit jobs to compute nodes. Gateway nodes are NAT servers that are located between compute nodes and the Internet. Compute nodes are not directly connected to the Internet, their access to the Internet is relayed through gateway nodes.

ABCI 3.0 provides a set of system software that makes maximum use of hardware resources and realizes advanced development of foundation models, and cloud operation. Table 2 shows the list of major software. Note that this is just a list at the initial installation time and will be updated in the future. In order to facilitate easy migration from the previous system, the usage model is basically the same, including operating system, job scheduler, a temporal shared file system using local SSDs (BeeOND), and container support. A notable additional feasure to the previous system is Open OnDemand [6, 7] support, which provides users with a way to access computing resources via a web browser instead of SSH.

In order to achieve high resource utilization, ABCI 3.0 uses Altair PBS Professional job scheduler for resource management. Each compute node has eight GPUs, but some workloads may require fewer GPUs and/or CPU cores. ABCI 3.0 allocates the same compute node to multiple jobs without resource oversubscription, with the job scheduler dynamically allocating a partial resources using cgroups and disk quota. Table 3 summarises ABCI 3.0 resource types. It intends a single compute node (rt_HF) can be divided into eight rt_HG instances and two rt_HC instances.

Unlike HPC, AI related software updates frequently, making it difficult for system operators to prepare all the necessary software in advance. Therefore, it is necessary for users to be able to freely customize their software environment. One such mechanism is environment modules, and the other is containers. We provide several environment modules to enable users easily use various kinds and versions of software libraries, including CUDA, cuDNN, NCCL, MPI, and Python. Users can build their development environments by loading environment modules and then installing necessary Python libraries such as PyTorch, DeepSpeed, Hugging Face Hub by running pip, Anaconda, or similar tools. There are several container runtime systems that integrates well with HPC environments. For example, Singularity, Podman, and Enroot. ABCI 3.0 initially support Singularity Community Edition (CE). In addition, Spack software package manager helps users compile and install software that is not officially supported by ABCI.

Figure 2 shows the overview of our AI data center facility that was built at the same time as ABCI 1.0. We have reported the detailed design of the original AI data center in [8]. For the installation of ABCI 3.0, we have upgraded the electrical and cooling capacities. Currently, we have 6MW of electrical capacity, 5.2MW of cooling capacity, and 144 racks of space.

ABCI 1.0 and 2.0 are very successful systems for advancing AI R&D in Japan. However, in the era of generative AI, AI industry is changing rapidly, and the situation surrounding ABCI is completely different from when we started the ABCI project. With government and other investment, the presence of domestic commercial cloud providers is growing. As a result of this situation, the role of ABCI must be shifted. ABCI 3.0 should focus on challenging research and development based on national demands, such as sovereign AI, and encouraging AI startups should also be an important mission. We provide large-scale computing resources needed for more challenging R&D by relaxing the current usage limits, including the number of nodes per job, node-time product per job, maximum reserved duration, and etc. We also plan to conduct development acceleration programs and user events. The latest information is available on the ABCI homepage [1].