Zeno: A Scalable Capability-Based Secure Architecture

The Zeno architecture uses an extended addressing model [10] to support the large addresses and the use of multiple Namespace address space abstractions simultaneously. Extended portions of addresses encode a Namespace ID to reference Namespace Metadata and identify which address space abstraction the base address references. Figure 1 shows the contents and layout of Namespace metadata. Namespace Metadata stores access permissions, in addition to information about the hierarchical organization of Namespaces. Minimum and maximum address bounds describe the Namespace’s accessible range of addresses in the address space abstraction. Read, write, execute, and valid permission bits describe the allowable memory operations. A physical page number of the Namespace page table is also included in the metadata. Each parent and child Namespace provides access to the same address space abstraction and therefore shares a page table. Metadata includes the Namespace’s Root Namespace ID, that is, the Namespace ID at the root of this hierarchy of Namespaces. A Root Namespace has no Parent Namespace and is useful for building address space IDs in the microarchitecture. Many Root Namespaces will exist simultaneously in a Zeno system. Though no current operations require it, the direct Parent Namespace ID is also stored in the Namespace Metadata to support system debugging and future operations that may reason about the Namespace hierarchy. A pointer to a list of children Namespaces is included in the metadata of each Namespace to support recursive Namespace Revocation. All Namespace Metadata is stored in the Distributed Namespace Directory, a dedicated Namespace accessible only to hardware and trusted firmware responsible for managing Namespace data.

The Namespace ID acts as an access token, granting access to Namespace Data. Namespace IDs are created by hardware for the requesting software context. Software is free to share its available Namespace IDs with other software executing anywhere in a multi-node system. Each node in the system views a shared Namespace as the same address space abstraction. Without the Namespace ID access token, software cannot access a Namespace. Namespace IDs are protected by tag bits in memory and on the processor die to prevent forgeries. The hardware-enforced address bounds of Namespace capabilities provide spatial memory safety. Namespace revocation supports temporal memory safety by removing all access permissions granted by a Namespace without the need to invalidate the Namespace ID access token everywhere in the system.

Zeno cores include an extended set of 64-bit registers to store Namespace IDs. Figure 2 shows the micro-architecture of a Zeno core. Signals and modules in blue depict logic added to a baseline RISC-V core to support Namespace IDs. Signals and modules in red depict the base rv64i signals.

Each core in a Zeno system must support Namespace creation, derivation, and revocation. Table I describes the inputs and outputs of each operation. In this work, the Namespace operations are each encoded into a single instruction to ensure they happen atomically. The extended register file is used to store Namespace IDs and pack up to four register reads into a single instruction while conforming to RISC-V’s standard encoding for register selection.

The NS_CREATE instruction sets the Namespace metadata in memory and writes a valid Namespace ID into the extended register file. To minimize changes to the processor datapath, the NS_CREATE instruction is decoded into several RISC operations to store appropriate metadata values in memory. The NS_DERIVE instruction is similar to the NS_CREATE, but also requires the Namespace ID of the Parent Namespace. The NS_REVOKE instruction requires only the Namespace ID to be revoked. If a Namespace with valid children is revoked, the system must recursively revoke them with either hardware/microcode operations, or a fault and trusted firmware.

The Zeno architecture does not require any changes to on-chip instruction/data caches, but does require additional Memory Management Unit (MMU) logic. To improve performance, Zeno includes an on-chip Metadata Cache inside the MMU to quickly access Namespace Metadata. When a memory access is issued by the Zeno core, the Metadata Cache uses the Namespace ID to search the cache for the associated Metadata in parallel with TLB lookup and the virtual address translation. On a hit, the metadata is read out to the permissions checking logic, which validates the requests and allows the memory request to continue. On a miss, the metadata cache can either raise a fault or fetch the metadata with logic similar to a page table walker. These options mirror the RISC-V address translation ISA specification.

In the MMU, the Root Namespace IDs are added to the TLB tags to distinguish between the different Namespace address space abstractions. We denote this TLB the Namespace-TLB (N-TLB). The page table walker is updated to use the Namespace Metadata page table physical page number rather than RISC-V’s SATP register.

Remote memory is accessed through a remote-memory cache stored in local DRAM, allowing remote data to appear local to software and address translation mechanisms. The remote data cache scheme allows hardware to fetch the remote data without any intervention from local software or remote CPUs, simplifying programming and improving performance. For scalability, remote memory cache coherence is managed in software. To support hardware fetching of remote data, each Zeno node includes a network interface for remote memory access. The network interface sends Namespace capability memory requests to remote nodes and serves request received from other nodes. The network interface is coherent with the Node’s local cache hierarchy to maintain coherence within a node. The network interface must perform the same Namespace permissions checks as the MMU connected to each core and VIPT L1 cache. A strait-forward network interface implementation re-uses the cache hierarchy typically connected to a core and connects it to a DMA engine serving requests across the system interconnect.

Zeno connects distributed nodes of compute resources together into a single system. Example node configurations include: a processor and memory, just memory resources, or I/O disk resources. Figure 4 shows an example four node Zeno system. Each node has a multi-core processor and globally accessible main memory. Zeno is intended to support large scale systems, i.e. many racks worth of compute and storage nodes. However, smaller rack-scale and single node systems can also be implemented. Zeno does not require a specific interconnect topology or protocol to connect multiple nodes together, as long as they are connected through Namespace validating Network Interfaces.

Figures 6(a) and 6(a) demonstrate the performance scaling of the Get Transfer and Integer Sort benchmarks on Zeno architecture as more nodes are added to the system’s global memory space. Figures 6(b) and 6(b) show the fraction of runtime spent constrained by each major subsystem in the architecture (the CPU, NLB, local memory hierarchy, and global memory hierarchy) for each benchmark. Both the Get Transfers and Integer Sort benchmarks are each simulated in a Zeno configuration with only 32 N-TLB and metadata cache entries to ensure the benchmarks working set could not be completely buffered in either structure. Systems are arranged in 2D meshes with sizes ranging from 2x2 to 8x8. IPC results are normalized to the IPC of a baseline 2x2 xBGAS [10] system with none of Zeno’s Namespace capability memory protections.

Figure 6(a) indicates that the performance of the Get Transfers benchmark remains steady for 4-36 node systems and decreases in the 49 and 64 node systems. To investigate the cause of this performance reduction, we plot the fraction of runtime dedicated to CPU execution, local and global memory accesses, as well as NLB lookups. Note that while the Get Transfers benchmark sees a reduced IPC for larger systems, this is not caused by an increased amount of global memory accesses or NLB lookups. Instead the additional latency stems from extra local memory accesses as the working set of the benchmark increases with the size of the system.

Figure 6(a) shows that the Integer Sort benchmark scales nearly perfectly with an increasing system size. This is because remote Namespaces are only accessed to set up the parallel bucket sort and collect the results. Figure 6(b) demonstrates that, although the Integer Sort runtime is dominated by CPU execution cycles and local memory accesses, the fraction of runtime spent on global memory accesses remains steady.

Taken together, Figures 6 and 6 demonstrate that the Namespace based access scheme forms a scalable system that will not be limited by the increased overhead of Namespace Metadata lookups. For the Get Transfers benchmark, the Metadata Cache lookup overhead represents 2-3% of the total program runtime. For the Integer sort benchmark, the Metadata Cache lookup overhead represents less than 0.1% of the total program runtime.

Next we examine the design trade-off between different MMU configurations with the Random Memory Access and Integer Sort benchmarks. Including both a cache for metadata and a Translation Lookaside Buffer creates a rich design space where it may be difficult to find an optimal size for each structure. A larger Metadata Cache and N-TLB will lead to greater performance but will also increase power and area requirements. Furthermore, it is not clear how a fixed area should be shared between the N-TLB and the metadata cache in the new Zeno architecture.

To explore the NLB design space, we simulate both benchmarks with metadata cache and N-TLB sizes of powers of two ranging from 4-128. This is six sizes per structure or 36 simulations per benchmark. Figure 8(a) and 8(a) depicts the Metadata Cache hit rates with a 32 entry N-TLB. Figure 8(b) and 8(b) depicts the N-TLB hit rates with a 32 entry Metadata Cache. A 16 node Zeno system is used in each of the simulations. Each benchmark uses 128 Namespaces. The trends in hit rates for increasing metadata cache and N-TLB sizes are similar. The hit-rates suggest that neither the metadata cache or N-TLB can dominate the MMU area, as performance steadily improves in both benchmarks with larger structures.

To gain more insight, we plot the overhead of additional global memory accesses in Figures 10 and 10 for the same benchmarks and MMU configurations used in Figures 8 and 8. The reduction in overhead in the Integer Sort benchmark as the NLB structures increase in size (Figure 10(a) and (b)) is similar, indicating that the metadata cache and N-TLB are equally important in the performance of the Integer Sort benchmark. However, comparing Sub-figures (a) and (b) in Figure 10 shows that the Random Memory Access benchmark benefits more from a large metadata cache than a large N-TLB. The better performance of the large metadata cache stems from the greater fraction of addressable memory covered by a full Metadata Cache than an N-TLB of the same size. For applications with better spatial or temporal locality, this difference is less pronounced. In the random memory accesses used here (but also in HPC applications such as graph processing) the impact of this improved global memory space coverage becomes apparent.

Finally, we examine the performance overheads of programming with many or few Namespaces. Using many Namespaces enables greater isolation and increased memory safety but at the expense of performance, as more Metadata Cache and N-TLB misses are inevitable. Figures 12 and 12 quantifies this trade-off by presenting the overhead of increased global memory accesses caused by metadata cache and N-TLB misses. Again, simulations are run on a Zeno system with 16 nodes where each NLB includes a 32 entry metadata cache and 32 entry N-TLB. The additional cycles spent accessing global memory are normalized against a plain xBGAS system that implements a flat 128-bit memory space without any security protections. The smallest test limits the number of Namespaces to the size of the metadata cache and practically eliminates the overhead of fetching Namespace Metadata. The initial 32 metadata misses are insignificant compared to the total program runtime.

Figure 12 shows that increasing the number of Namespaces from 32 to 128 in the Random Memory Access benchmark leads to a 1.4x overhead. The overhead of going from 32 to 128 Namespaces in the Integer Sort benchmark, shown in Figure 12, is higher at 4.1x. The relatively low overhead of the Random Memory Access benchmark is likely because of the limited effectiveness of the Metadata Cache and N-TLB with a random access pattern in the first place. Since there was little spatial or temporal locality for the caches to exploit at 32 Namespaces, frequent global memory accesses were required to fetch Namespace Metadata and address translations from memory. Utilizing 128 Namespaces instead of 32 did not significantly worsen any locality in the benchmark since there was little to begin with. The Integer Sort benchmark has more spatial locality to exploit, especially at the end of the computation when results are collected on a single node. Figure 12 shows that the overhead of additional Namespaces can become more significant. However, the additional overhead begins to level off as more Namespaces are included in the program, indicating that spatial or temporal locality in the program allows the Metadata Caches and N-TLB to remain effective.

Table III presents synthesis results for the Zeno architecture, an xBGAS enabled processor without Zeno’s additional micro-architecture features and a baseline RISC-V system. Each system includes includes a seven stage pipelined rv64ima core with 32kB, 8-way instruction and data L1 caches, a unified 256kB 8-way L2 cache and Memory Management Unit. The xBGAS-enabled system and the Zeno architecture extend the baseline RISC-V core with support for the xBGAS ISA extension. Additional micro-architecture features of the Zeno architecture include the Metadata Cache, modified TLB and permission checking logic. The Zeno system is configured with a 1024 entry, 8-way set associative N-TLB and a 128-entry, 8-way set associative Metadata Cache. The targeted FPGA is an Altera Stratix V (5SGXEA7N2F45C2) [12]. Quartus II 15.0 is used for synthesis. Area results for Adaptive Logic Modules (ALMs) [12], registers and BRAM bits are reported. The percentage of FPGA ALMs and BRAMs used is reported in parenthesis next to the raw number of resources used.

Most of the resource overhead necessary to implement the xBGAS ISA extension is dedicated to the additional register file, with instruction decode logic and a small number of additional pipeline registers making up the remainder of the overhead. The Zeno architecture adds a 9.8% overhead in FPGA area (ALM Utilization) and a 5.4% overhead in total registers relative the baseline RISC-V rv64ima system. The synthesized configuration has an 4.3% BRAM memory overhead. The absolute difference in area is 4,459 ALMs, representing approximately 2% of the total FPGA area. The Zeno core and memory hierarchies achieve an Fmax of 130.2MHz, compared to 131.6MHz for the baseline BRISC-V core and cache hierarchy.