The Locus Algorithm III: A Grid Computing system to generate catalogues of optimised pointings for Differential Photometry

Access to the grid was provided by means of a dedicated computer called gridUI (grid user interface), which acted as a gateway to the grid, located at ITTD. This computer used the same operating system and shell, and had the same access to the grid as a grid node and as such was ideal for testing and debugging grid software. In addition, it was on gridUI that grid jobs were generated, and from which jobs were submitted and monitored, and finally to which results were downloaded when needed.

The long-term, large-scale distributed file stoage system used in NGI is the LFC (Coghlan et al., 2005). The LFC system consists of three structural components (Sciaba et al., 2012).

The first component is a catalogue of Logical File Names (LFN) linked to their corresponding Globally Unique Identifiers (GUID) and Storage Universal Resource Locator (SURL) (Kunik, 2006). LFNs are designed to be human-readable and are structured to mimic a standard UNIX File System (UFS). GUIDs are a unique string of the form guid uniquestring and are used as a primary key for each file in the catalogue. SURLs refer to the location where the file is physically stored.

The physical storage referred to by the SURL is the second element of the LFC suite (Kunik, 2006; Sciaba et al., 2012). Unless the user specifies otherwise, the physical location of files is assigned automatically by the GMS and is distributed across the various locations of the grid system.

Data in the LFC cannot be directly accessed, but rather must be accessed by means of the final element of the LFC – the set of gLite middleware commands used to interact with the data in the LFC (Sciaba et al., 2012). These commands are designed to mimic the UNIX shell commands. For example, the gLite command lfc-ls will list the contents of a directory specified by its LFN in the same way that the ls UNIX command will list the contents of a directory in the UNIX file system (Sciaba et al., 2012).

Similarly, the user does not interact with the grid nodes directly. Work is instead assigned to the nodes by the JSS (Sciaba et al., 2012). Users submit individual tasks to the JSS by means of grid jobs. Grid jobs are each defined by a single file written in the Job Description Language (JDL). .jdl files consist of a series of name-value pairs which fully define the task assigned to the grid and specify any requirements that that task may have such as available storage, memory or permitted job time (Sciaba et al., 2012).

A .jdl file must include the path to an executable file which will be copied from gridUI to the WN and executed on the WN (Sciaba et al., 2012). Additional requirements may include data files that are required for the job (Sciaba et al., 2012). Unless the user specifies otherwise in the job, the physical location of the worker node is selected automatically by the Workload Management System (WMS). The WMS is designed to handle load balancing and job scheduling and thus ensure that WNs are never left idle.

Jobs are submitted to the GMS using gLite commands. When these commands are executed, the JDL file is sent to the JSS, and the executable file, together with any data files specified in the job, are uploaded to the GMS (Sciaba et al., 2012; Kunik, 2006). These files are then downloaded to the working directory of the WN to which the job is assigned. This is an essential consideration for the design of the executable software.

Further gLite commands exist for the monitoring and management of grid jobs (Sciaba et al., 2012; Kunik, 2006). Grid jobs, once submitted to the WMS are given a status to indicate their progress from “submitted” to “cleared” which may be monitored by the user manually, or automatically as used in this project and discussed in Section 5 (Sciaba et al., 2012).

Grid jobs may fail to complete for a variety of reasons, including grid problems, such as node crashes or software problems such as memory leaks or missing data. The GMS does not automatically resubmit failed jobs, instead requiring that the user monitor the progress of jobs, and resubmit them if appropriate. Checkpointing may be implemented by the user using the LFC and suitable scripting techniques if desired, but is not part of the default system (Sciaba et al., 2012).

The NGI was a system physically distributed at several widely separated sites connected via WAN. The Workload Management System monitored the storage and compute elements at these locations, and distributed the workload appropriately. Unless the use of specific locations was specified by the the user either directly or by specifying distinctive attributes of the nodes, the WMS would automatically select a location for LFC storage or jobs submitted through JSS. Users did not need to specify locations.

The Parameterisation software is part of the job management system responsible for partitioning an overall task into jobs, each of which consists of a number of work units. These work units are the smallest unit of work which can be carried out by a given element of the software: they are different for each mode of operation. For the API, which is responsible for extracting data from the Source Format into the Local Format, the work unit is a Source Catalogue file. For the Pipeline, which determines the optimum pointing for a list of targets, the work unit is a single target when operating in Target List mode and a Local Catalogue file when operating in Catalogue traversal mode as detailed in in Subsection 6.2. A job is a collection of work units bundled together such that they can be passed to the appropriate program and it will be able to carry out its task. The number of work units per job is selected by the user when running the parameterisation software and is optimised to ensure the task will fit the processing requirements of the grid system as discussed in Creaner et al. (2020c).

The Parameterisation software is designed to provide flexible input to the API and Pipeline programs. Early prototyping on these programs revealed that they would become heavily dependent upon SDSS file and directory structures without a layer of data abstraction to allow this information to be passed in by means of a parameter file. The parameterisation software generates two types of binary parameter file, PRM files for the API, and PPR files for the Pipeline. These files contain explicit file paths, target lists and control variables for the API and Pipeline programs. Because of this, the software can be applied to other catalogues with changes to the parameterisation software, but minimal changes to the other programs.

When developing the bash shell scripts used for grid management, it was determined that the PPR and PRM formats cannot easily be interpreted by the shell. As a pragmatic solution to this problem, text files containing the file paths were implemented which are accessed by the shell scripts, as a supplement to the binary files used by the C programs. These files, though stored with the file extension .txt, are referred to as Parameter Text (PRT) files in this paper to distinguish them from ordinary plain text files.

The primary component of this system is a program called parameterisation. This program takes as input a combination of command line arguments and data stored in CSV files of various structures . The output of this program is a number of parameter files, the size and structure of which is determined by command line arguments which designate which mode of operation the program employs. It operates in three modes: API, Target List Pipeline and Catalogue Traversal Pipeline as illustrated in Figure 5. Based on the input command line parameters and the contents of the CSV file, the program automatically determines which mode it will operate in.

In all modes, parameterisation takes the following common arguments: the number of work units to include in each grid job and three paths in which to: (1) store output parameter files (2) find the CSV file containing data to parameterise (3) find the FITS files the parameter file will refer to. For both pipeline modes, the user must supply observational parameters regarding the telescope the output is intended for use with. These parameters are FoV size (in degrees), Resolution (in degrees), Maximum magnitude difference (in magnitudes) and Maximum colour index difference (in magnitudes). Finally, to operate in Catalogue Traversal mode parameterisation requires a PRM file which contains the paths to the files for each of the target fields. These parameters are summarised on Table 1.

The grid was used for four operations within the scope of the project. Extracting data from the source Catalogue to the Local Catalogue through the API, calculating pointings for a list of quasar targets to generate the quasar catalogue as shown in Creaner et al. (2020b), generating PPR files for catalogue traversal as discussed in Subsection 5.1.3 and calculating pointings for all targets in the Local Catalogue as required for Creaner et al. (2019).

All grid operations require grid management software, written in Bash to operate. For each of these operations, the same four components of the Grid Management Software (GMS) exist: job generation (see Subsection 5.2.1), job submission (see Subsection 5.2.2), job calling (see Subsection 5.2.3) and job monitoring (see Subsection 5.2.4). The design of these components is largely identical across each of the four operations. The differences are treated here as minor variations to the design and highlighted where significant.

The Locus Algorithm is designed to be flexible with regards to source data. By extracting data from a Source Catalogue into a Local Catalogue, the data required for the algorithm is retained, while other data is discarded. The Local Catalogue format was designed such that data from any catalogue containing position (RA, Dec) and magnitude information could be extracted into that format and used with the Locus Algorithm.

The source catalogue for this project was the SDSS Catalogue of Calibrated Objects. This catalogue was held on the SDSS Data Archive Server (DAS) (Alfred P. Sloan Foundation, 2007) in a collection of many Calibrated Objects files (tsObj files). This catalogue was downloaded to the LFC and stored there for processing. These files each contain 146 columns, and each row refers to an observation of an object by an SDSS camera. Only a few of the attributes recorded in those 146 columns are required for the software system presented here. Many of these records are of non-stellar objects not suitable for use as reference stars. In addition, due to the observation pattern used by SDSS, many of these records are not “primary” observations as defined in the SDSS Image processing flags (Alfred P. Sloan Foundation, 2006).

The SDSS Data Access API is designed to extract data from the SDSS catalogue in such a way as to minimise the data volume that is accessed in the pipeline by both projection and selection operations on the source catalogue files. The selection operation is carried out by filtering to only those records which meet the SDSS clean sample of stars criteria, which discards non-stellar and non-primary sources (Alfred P. Sloan Foundation, 2006). The projection operation consists of selecting only the position (RA, Dec) and magnitude (ugriz) columns from the data as the other columns are not required for the operation of the Locus Algorithm (Creaner et al., 2020a).

The API is split into three conceptual blocks as illustrated in Figure 7: work carried out on gridUI, work carried out on the WN, and interactions with the LFC.

The first block consists of parameterisation as described in Subsection 5.1.1, where user input is given to the parameterisation system to determine how many work units (SDSS fields) to assign to each grid job, which are then listed and stored in paramter files on gridUI. For each of these parameter files, a JDL file is generated by the generate_jobs_api script which creates a series of .jdl files. These files are submitted to the JSS, which assigns each job to a WN.

call_api operates on a WN, and takes two parameter files as inputs from gridUI. One is in PRT text format and one is in PRM binary format. The PRT file specifies the LFN paths in the LFC to a set of input SDSS fits files which are to be processed by the API to create the corresponding local catalogue files and is used by the shell script. A for loop iterates through this file, and uses the lcg-cp command to copy the listed files from the LFC to the WN.

Two programs, Diagnose and Extract, are then copied from the LFC to the WN. These programs use the PRM parameter file as an input to access the fits files. The Diagnose program accesses and analyses the data contained within the input files, and identifies the columns in the data table. The column names together with information regarding their structure are then stored in a file. Next, the Extract program carries out the substantive work of the API. Taking the columns identified by Diagnose, it identifies the columns containing only the relevant data: Right Ascension, Declination and Magnitude (itself an array of 5 double values). It then applies the SDSS clean sample of stars algorithm to exclude entries which are not primary entries for stars(Alfred P. Sloan Foundation, 2006). The data for the remaining entries in those three relevant columns for this project are then written to a file which forms part of the local catalogue.

Finally, call_api then uses lcg-cr to copy all of the Local Catalogue files to the LFC for storage. These files are then used as the inputs to the Pipeline

The Data Pipeline applies the Locus Algorithm as defined in Creaner et al. (2020a) to a set of targets to produce output files which form the Output Catalogue(s). It operates in two modes: Target List and Catalogue Traversal. Target list mode is used when a set of one or more targets are selected in advance by the user and are submitted to the pipeline for processing. This mode was used to produce the Quasar Catalogue as discussed in Creaner et al. (2020b). Catalogue traversal mode uses an existing catalogue or subset of a catalogue to produce the target list. This target list is then submitted to the pipeline in a similar way to the Target list mode discussed above. This mode was used to produce the Exoplanet Catalogue as presented in Creaner et al. (2019). The structure of these modes is outlined in Figure 8.

Much of the software is designed to ignore the distinction between these two modes, for example, by treating a single target as a list of length one. Distinctions between operational modes are therefore discussed only as needed, as shown in Figure 8, where dashed boxes are used to indicate modules that differ between the two modes.

In Target List mode, a target list consisting of the positions (RA, Dec) of a set of specific targets (e.g. quasars) is submitted through an SQL script to the CAS (Alfred P. Sloan Foundation, 2017) together with FoV size. For each target in the list, the script identifies which fields include any objects which are inside the Candidate Zone (CZ) as defined in Creaner et al. (2020a). The CAS returns a list of field identifiers as required for parameterisation as described in Subsection 5.1.2. This list of identifiers is combined with observational parameters and passed through the parameterisation and generate_jobs scripts as defined in Subsections 5.1.2 and 5.2.1 to generate a set of grid jobs.

In Catalogue Traversal mode, a field list, consisting of the field identifiers for all fields in the catalogue is submitted to the CAS instead of a target list. The SQL script identifies, for each target field, the neighbouring fields which have at least one object within the CZ for any target in the target field. The target fields and reference fields are then passed through the parameterisation and generate_jobs scripts as above.

Each grid job calls a script named call_pipeline which uses the parameter files to identify the necessary local catalogue files and generates the output catalogue based on the listed targets. Two versions of the Call Pipeline script execute the pipeline, one in each of the two modes. The primary distinction between these modes is that when run in Target List mode, the Parameter files are stored on gridUI, while in Catalogue Traversal mode those files are stored in the LFC. This means that in the first case, the job submission process submits the PPR files as part of the grid job, while in the second, they have to be copied out of the LFC using gLite commands. The two versions are otherwise identical as shown in Figure 8.

For each Local Catalogue file listed in the PRT file, call_pipeline checks to see if the file is already present on the WN, and if not, copies it from the LFC to the WN. Note that each file is likely to be used by more than one target in the target list, especially in Catalogue Traversal mode.

Call pipeline then executes the C program locus_algorithm, defined in greater detail in Nolan et al. (2020). This program with the colour argument provided in the JDL file, redirecting its output to /dev/null unless operating in verbose mode for debugging purposes. This program implements the Locus Algorithm and generates an output file.

Finally, the output file is copied and registered to the LFC and any errors are reported to the Grid Management Software.

While the system was implemented with the SDSS Catalogue, it was developed with flexibility as to the source of the data. This flexibility was enabled by the use of the Data extraction API described in Subsection 6.1. This system extracted the data that was required for the algorithm from the source catalogue and processed it into a new structure called the Local Catalogue. The Local Catalogue consisted of a series of FITS files which could be read by the main data pipeline.

In future iterations of this system, it is envisaged that a new API could be developed to read data from the structure of other catalogues (e.g. Gaia) and write it to the Local Catalogue format. This abstraction provides a layer of source-independence for the processing pipeline

As a practical matter, the directory structure of the local catalogue mimicked the structure of the source catalogue. However, the local catalogue directory structure was abstracted from the pipeline software through the parametrisation software as shown in Subsection 5.1. This software queried the CAS to identify targets (or target fields) and the reference fields that would be needed, and provided the pipeline with the paths to those fields. This parameterisation software can be updated in a modular fashion to allow it to work with new catalogues.

Because of these abstractions, the core software of the pipeline is independent of the structure of the source catalogue, depending instead only on the local catalogue.

Data in this project was stored in a variety of formats to fit with the requirements of the sources and the systems that were employed. These are summarised in Table 3, and discussed in greater detail in Nolan et al. (2020).

Data in the Source Catalogue was in the Flexible Image Transport System (FITS) format, a widely used format in the astronomical community. To minimise the number of dependencies of the software system, it was decided that the Local Catalogue and the Output Catalogue would also be stored in FITS format.

The parameterisation software which formed part of the job management and data abstraction elements produced data in two novel formats developed for this project. PRM files were used to identify files in the source or local catalogue without structure for grouping. PPR files were used to identify targets for processing in the pipeline, grouped together with files to be used in groups to form mosaics around those target(s). PRM and PPR files are binary format files which are interpreted by the core software. PRT files are text files including the lists of target files from PPR or PRM files which can be interpreted by the shell scripts used for grid job management as discussed in Subsection 5.

CTI (Catalogue information) is a file format which was built for this project to enable information about the structure of catalogue files to be stored in a lightweight, accessible format as part of the API as discussed in Nolan et al. (2020).

The Job Description Language (JDL) is used to define grid jobs as part of the glite system (Sciaba et al., 2012). JDL files are automatically created by the generate_jobs scripts which access the list of parameter files for a given grid operation and create new JDL files which can be submitted to the JSS as discussed in Subsection 5.

Comma Separated Value (CSV) files are used in two ways in the project. First the outputs from the CAS are delivered in this format. The CAS supplies the lists of fields in the source catalogue which are used in the parameterisation software to define the paths to the files used in the API. The CAS also supplies lists of targets and their associated fields which are used by the parameterisation software to generate the jobs for the pipeline. In addition, the output catalogues, which were created in FITS file format, have also been converted into CSV format for publication, for the benefit of users who do not use FITS.

Over the course of this project, data was created, stored and used on a variety of different storage elements, some of which were volatile and others were persistent relative to the timeline of the project. Data transfer between these elements was a key design consideration, as network data transfer operations are slow by comparison to local file I/O operations. A schematic of these data storage elements is presented in Figure 9

All user interaction in the project was carried out on the Development Environment (DE). This was a local Windows-based system with virtual environments configured to reflect WNs. This allowed for rapid development and testing of software independent of grid operations. This system was connected to gridUI through Secure SHell (SSH) which allowed both file transfer (for software) and user input to be communicated to the grid.

As described in Subsection 3.1, gridUI was the gateway to the grid and had access similar to that of a WN. Programs, parameter files and grid jobs were staged here for submission to the grid. The glite system was used to submit these elements. Programs were transferred to the LFC through the GMS by submitting lfc-cr commands. JDL files were created on gridUI and submitted to the JSS. The JSS would then assign the job described in the JDL file to an available WN and monitor its progress. The user could submit requests to the JSS using the job monitoring software described in Subsection 5.2.4 to track the progress of a grid operation.

As described in Subsection 5.2.3, grid jobs typically included lfc-cp commands to transfer programs and the required data from the LFC to the WN to allow a job to begin, and lfc-cr commands to transfer output data to the LFC when the job was complete. The data abstraction described in 7.1 above greatly reduced the volume of data to be copied from the LFC to the WN.