Modern Software Development for JUNO offline software

CMake macros and functions have been developed to put all the common functionalities in the same place. As there are some existing conventions defined in CMT, some of them are used in the CMake macros. According to the instructions in Modern CMake, the following rules have been used in the software development with CMake:

• •

  The source code, build directory and installation directory of a project are separated. Even though CMT adopts a similar rule, CMT puts all the build directories under the package directories. When moving to CMake, they are all separated to keep the source code clean. An example is the source code generation for the event data model. When using CMT, the files are generated in the source code directory, which causes the check-in by mistakes sometimes. After moving to CMake, these files are generated under build directories.

• •

  A project is organized into packages. A package consists of a header directory for public interfaces, a source directory for the private headers and detailed implementation, a python directory for exporting the library in Python, a share directory for the regularly used scripts, and a test directory for the testing scripts.

• •

  A CMake target is used when building a package. It is used to represent a shared library, a module library or an executable. Its dependencies on the other different packages are described by the other CMake targets. The CMake target properties are used to control how a target is built instead of using global settings.

• •

  CMake generator expression is used to control the flags instead of using the if statement in CMake. This is useful when the same package is built for online and offline environments. In this case, the linking libraries could be different. By using the generator expression, the libraries could be enabled or disabled by checking an option defined in CMake.

• •

  Macros PKG and EDM are developed to build a regular package and a package containing event data model respectively. The macro EDM generates C++ source code and ROOT dictionaries at the CMake configuration stage, and builds a shared library at build stage. The macro PKG creates a shared library by default. If a module library needs to be created, then an option MODULE is needed. If there is no library or executable created, a custom target will be created to install the python and share directories.

• •

  Environment variables of packages are collected in the macros PKG and EDM. By adding package names to a global property in the project, all the information of a package could be accessed, including the environment variables. A CMake script is used to create both bash and tcsh scripts before installation. All the environment variables are added at the end of the scripts.

• •

  When a project is installed, a CMake config will be created automatically, including all the targets within the same namespace. Another project needs to use the CMake config file to locate the project and load the exported targets.

• •

  The CMake commands and options are put in the build.sh script.

Below is an example of CMakeLists.txt for the package Geometry:

In this example, the PKG declares the package name. As there are no explicit files to be compiled, all the files under the source code directory will be used. As there is no option MODULE, a shared library will be created. When compiling and building this package, it will depend on several libraries, which are defined after option DEPENDS. As mentioned before, the target names are used. Both Identifier and Parameter are from JUNOSW, while the others are from external libraries. The target Parameter is not used if the software is built for online.

Below is another example of the build script, which consists of three steps:

As Git is already widely used in the HEP community, the migration to Git is not so difficult. After the migration is done, users request to support the partial checkout and build. This is common when using CMT. In order to support partial checkout, git sparse checkout is used. For the partial build, the CMakeLists.txt is set up with customized build targets.

In order to support the customized build targets, users are allowed to provide their own file, named CMakeLists.user.txt. This file could be edited by users, or controlled by the git-junoenv tool. When using git-junoenv to check out packages partially, these package names will be registered into CMakeLists.user.txt automatically. This could also used to build packages for online environment. Below is an example:

After both partial checkout and build are working, a shell script called git-junoenv is created. By prefixing git in the command name, this command could become a sub-command of git. Below is an example when using it:

When users need to develop with JUNOSW, the first step is using init-project to clone the code from the official repository. In order to hide all the packages, both sparse and no-checkout options are used during the git clone. After cloning, the script will check out the CMake-related code and initialize the CMake file of users. Then, users could list all the available packages in the project. The command git ls-files is used to list all the directories containing CMakeLists.txt. Users could use add-pkg to checkout and enable the package.

The installation script junoenv is inspired by the ENV project [10], which collects all the necessary installation scripts in the same repository. There are more than 50 scripts for building external libraries, and about 30 libraries are deployed in the official release. Modularized bash functions are used to describe the metadata of the external libraries. When installing a package, the script junoenv loads the metadata of the package and drives the installation of it. There are five steps defined during the installation:

• •

  get: download the source code by cURL, wget or git;

• •

  conf: configure the package;

• •

  make: build the package;

• •

  install: install the package;

• •

  setup: create setup scripts for both bash and tcsh.

Five corresponding common bash functions are in charge of these steps. If additional configuration is needed, the package can define its own functions to override the default functions. Following is an example:

This is used to configure the package CMake. As the CMake does not use the configure script by default, an additional function suffixed with a dash is defined to override the default behavior. So the invoking procedure is: junoenv invokes the conf function of CMake, named juno-ext-libs-cmake-conf; then this function invokes the common function, named juno-ext-libs-PKG-conf; the common function then invokes the overridden function.

Reproducible is important during the deployment. For a dedicated release of JUNOSW, the versions of all external libraries need to be recorded. In order to track all of them, a shell script is used to collect them. When deploying a release, the corresponding script is loaded. The shell script self is created by invoking the vlist command, which prints all the installed packages and their versions. Below is an example of this script:

All the software is deployed into CVMFS. When deploying the software, the installation prefix could be different from the original one when building software. For most packages, it is not an issue. However, there are still several packages that hardcode the paths, such as physics generators. Inspired by the package manager spack [11], the paths during building and deployment are set as the same length, and they are replaced using the tool sed when deployed into CVMFS.

Continuous integration (CI) is one of the important parts of software development. If building JUNOSW starts from the external libraries, it will take quite a long time. In order to reduce the CI running time, the external libraries should be pre-installed. Two types of Docker images are explored:

• •

  Lightweight image with CVMFS clients installed. The image is built based on the CentOS 7. The size of the Docker image after compression is 372.87 MB. This is useful for both CI and developers with good network connections. It is required to run the privileged Docker container so that the CVMFS client can work correctly.

• •

  Full image with all the external libraries installed. There are two flavors: one is based on CentOS 7 with a size of 17.29 GB; another is based on Ubuntu 22.04, with a size of 20.37 GB. No special privilege is needed to run the container.

The lightweight image is chosen in the GitLab CI. The CVMFS client needs to be available before setting up the JUNOSW software environment. Below is the YAML configuration file:

The benefit is that the Docker image could be reused when upgrading the external libraries. In the above example, only the JUNOTOP needs to be updated in the YAML file.

GitLab runners are in charge of the execution of the CI jobs. They need to be deployed and associated with the GitLab projects. As there are multiple projects, GitLab group runners are set up in a self-hosted Kubernetes cluster. GitLab agents are used to connect Gitlab and Kubernetes. Then the Gitlab runners are managed by the Gitlab agents. All the configurations are managed in GitLab repositories.

The GitLab agent is set up as below.

Then GitLab runners are installed with the cluster management project, which is a Git repository. The configuration of GitLab runner is enabled first, then the runners will be set up automatically.