Distributed Compressed Sparse Row Format for Spiking Neural Network Simulation, Serialization, and Interoperability

The field of neural computing has witnessed significant developments and expansions in the software frameworks, network simulators, hardware platforms, and engineering tools available to the community (Schuman et al., 2017; Dai et al., 2020; Stimberg et al., 2019; Aimone et al., 2019; Intel, 2022; Rothganger et al., 2014; Davison et al., 2009; Brette et al., 2007). Especially in the software space, there has also been a continual push toward larger scale experiments in line with the advancements in high performance computing (HPC) and use of accelerators such as graphics processing units (GPU) (Golosio et al., 2021; Knight and Nowotny, 2021; Pehle and Pedersen, 2021; Wang, 2015). However, alongside this expansion of frameworks and tools is the question of their compatibility and interoperability within a shared ecosystem. For example, a common thread of discussion is how to adequately compare and benchmark between different simulators and target neuromorphic platforms (Vineyard et al., 2019; Kulkarni et al., 2021; Davies, 2019).

Recent efforts such as Fugu have attempted to resolve the interoperability between a spiking neural algorithm and different neurormophic backends by employing only the most widely supported neuron and synapse models (Aimone et al., 2019). In the adjacent machine learning (ML) community, there are also efforts such as the open neural network exchange (ONNX), which provides an open format built to represent ML models (e.g. between PyTorch and TensorFlow) (Bai et al., 2019). Perhaps most closely related to this work is the Scalable Open Network Architecture TemplAte (SONATA) data format for large-scale network models (Dai and et al, 2020). Although it leverages common file formats (e.g. csv, hdf5, json), we noted some concerns due to its use of hierarchical, population-based grouping (for both neurons and synapses), especially with regards to data locality.

In this paper, we propose the use of a relatively straightforward data format for interoperability and sharing of SNN models between simulators and neuromorphic hardware platforms. We base this on the widely used compressed sparse row (CSR) format for efficiently representing sparse matrices, and extend it to accommodate parallel partitions of network state. We first provide a high-level description of this format in Section 2, a pointer to an implementation in Section 3, and finally a discussion in Section 4.

Compressed sparse row is a widely used format to efficiently represent sparse matrices (Saad, 2011). Although there are several implementations, the central idea is to store the non-zero values of a matrix corresponding to indexical arrays over the rows and columns. For an $(n\times n)$ matrix with $m$ non-zero entries, the row array is of size $n+1$, where each entry provides a prefix or cumulative sum over the number of non-zero column indices for that row, and the last entry in the row array is $m$. The column array is of size $m$ and contains the indices of the non-zeros entries for a given row-column pair. The corresponding value array is also of size $m$ and simply concatenates the non-zero values as read off in row-major order.

As an extension to this, the distributed CSR (dCSR) format was most notably introduced for storing and ingesting large graphs in prior work in parallel graph partitioning algorithms (Karypis and Kumar, 1998). This format provides an additional indexical array of size $k+1$ which supplies the prefix or cumulative sum over the number of non-zero indices for a given $k$-way partitioning of the rows, where $n_{1}+n_{2}+\dots+n_{k}=n$. Furthermore, the original CSR value and column arrays are split into multiple arrays along these partitions and $m_{1}+m_{2}+\dots+m_{k}=m$.

To compare, for an SNN, a natural representation of the connectivity and overall network structure is as a directed graph (West, 2001). Vertices $V(G)=\{v_{1},v_{2},\dots,v_{n}\}$ correspond to the spiking neurons, and edges $E(G)=\{e_{1},e_{2},\dots,e_{m}\}$ correspond to the synaptic connections. The direction of an edge $e_{l}:v_{i}\to v_{j}$, for $i,j\in 1,\dots,n$ and $l\in 1,\dots,m$ corresponds to the propagation of an event (e.g. a spike) from the presynaptic neuron $v_{i}$ to the postsynaptic neuron $v_{j}$. We may further partition this graph by splitting up the vertices such that $|V_{1}|+|V_{2}|+\dots+|V_{k}|=|V|$, known as a $k$-way partition. Here, edges may be assigned to a given partition based on their source vertex or their target vertex. With respect to the communication and computational patterns for SNNs, typically with synaptic weights applying current on their target neuron, colocating a directed edge with its target vertex is more sensible.

We can see that there is essentially an isomorphism between the rows and columns of a sparse matrix to the vertices and edges of an SNN graph. The main difference is that for an SNN, the vertices and edges typically carry far more additional information (e.g. connection delays, neural and synaptic states) than may be afforded by the standard, single non-zero entry in the value array. To address this, we suggest the extension to simply allow for tuples of values to be associated with the column array (as well as tuples of values to be associated with the row array). Because the amount of necessary unique state for any given vertex or edge will depend on its specific model dynamics, we may also introduce an additional model dictionary to provide tuple sizes.

Transitioning from this high-level description to a more concrete implementation is actually fairly straightforward. In fact, we may simply extend the dCSR format such as used by ParMETIS (Karypis and Schloegel, 2013). This has the added advantage that we may directly interoperate with existing graph partitioning tools. Although generally less memory efficient on-disk than in simulation, we also opt to serialize to plain-text files for portability.

Instead of serializing a graph adjacency as two contiguous arrays of row offsets and column indices, respectively, one of the shortcuts implemented by ParMETIS is to implicitly index the row by its line number in a given .adjcy.k (adjacency) file and then list its column indices as space-separated values. Because the entire file must be read in from disk for processing, the initialization of data structures such as the row offsets can be computed at runtime. The partitioning offsets are still needed, however, and these are supplied through a separate .dist (distribution) file.

As additional information to a geometric partitioner, we also initialize a .coord.k (coordinate) file corresponding to the spatial coordinates of a vertex $(x,y,z)$ within a cartesian coordinate system. This becomes especially useful when network sizes exceed the memory requirements for advanced partitioners and may need to fall back to simple voxel-based partitioning.

For the main network .state.k (state) files, we supply a space-separated list of string-based model identifiers and tuples of state data. We opt to colocate the vertex and edge models together, resulting in a file that begins with a vertex identifier and its state, and followed by edge identifiers and state for each of the incoming connections. Of note here is that because the adjacency file for graph partitioning is typically undirected as opposed to directed, we additionally include special ‘none’ model identifiers with no associated state for edge where there is an outgoing connection but no incoming one. As mentioned previously, we also introduce a .model file which provides a mapping between the string-based model identifiers and the size of its state tuple, as well as shared model parameters.

There are also .event.k (event) files which provide serialization of any simulation events “in-flight” that have not yet been processed on the target vertex due to connection delays. These include space-separated tuples of the source vertex, arrival time, the event type, and any associated data.

Here, we point to an implementation of this distributed file format in the Simulation Tool for Asynchronous Cortical Streams (STACS) simulator (Wang, 2015). Incidentally, STACS was built from the ground up for parallel simulation using the Charm++ parallel programming framework which essentially forces serialization for the packing-unpacking of messages between parallel objects to support fault-tolerant computing (Kale and Krishnan, 1993). In effect, this enabled the decoupling of the network generation process and the simulation process through an intermediate serialized representation. It also served as an efficient format to snapshot of the network state for checkpointing-restarting and offline analysis.

As a comparative scalability example, we built and serialized the cortical microcircuit model consisting of roughly 76K neurons and 0.3B synapses (Potjans and Diesmann, 2014), resulting in about 12GB on disk (regardless of the number of partitions). For a 2x (in neurons) for 154K neurons and 1.2B synapses, our result was about 49GB on disk. This is effectively linear cost in number of synapses.

What makes the extended dCSR format particularly appealing is that it draws from pre-existing, widely used formats that are intuitive to understand. For our implementation, all of the network state is serialized into essentially four main types of parallel files (adjacency, coordinate, state, and event), and there is also little overhead in storage costs with the introduction of few additional metadata files (dist, model). Due to its simplicity, it also becomes relatively straightforward to interoperate with popular graph analysis packages such as NetworkX and its directed graph data structure (Hagberg et al., 2008).

Beyond its simplicity, we also contend that a partition-based distribution of network state makes dCSR more immediately suitable for computational parallelism, whether its target platform is between different nodes for simulation or between different chips on neuromorphic hardware. Furthermore, as a result of its lineage in graph partitioners, such a serialization may also be readily used to inform a potential repartitioning of an SNN model such that it may optimally fit to different backends. For these reasons, we put forward the extended dCSR data format for adoption within the neural computing community.