Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration

Caching and tiering share the same goal – to ensure most accesses occur at the performance layer. But, they differ in how they manage data, in the form of pages within an OS. Caching relies on page replacement algorithms to detect hot (or cold) pages and accordingly populate (or evict from) the performance layer while tiering migratespages between the performance/capacity layers based on the observed page access pattern (hot or cold)data resides exclusively in one of the tiers but not both. Essentially, caching operates in an inclusive page placement mode and retains pages in its original location, merely their original locations, only temporarily storing a copy in the performance layer temporarily tier for fast access. Conversely, tiering operates in an exclusive mode, actively relocating pages across various memory/storage mediums.

CXL memory (or PM ), with high throughput and low latency, is often treated as an extension of local DRAMFrom the perspective of OS memory management, CXL memory or PM appears to be a remote, CPUless memory node, similar to a multi-socket Non-Uniform Memory Access non-uniform memory access (NUMA) nodes. Hence, a tieringbased approach is commonly used to manage such a memory hierarchy. Indeed, state-of-the-art node. State-of-the-art tiered memory systems, such as TPP [Maruf_23asplos_tpp], Memtis [memtis], Nimble [Yan_19asplos_nimble], and AutoTiering[Jonghyeon_21fast], unanimously all adopt tiering to exclusively manage data on different memory tiers, i.e. , via dynamically migrating pages between fast/slow tiers to ensure efficient access to faster local DRAM .

However, CXL-based tiered memory has its unique characteristics. CXL memory dramatically increases the capacity Unlike the traditional two-level memory hierarchy involving DRAM and disks, in which DRAM acts as a cache for the much larger storage tier, current CXL memory tiering treats CXL memory as an extension of local DRAM, accommodating a broader array of applications with data distributed across both fast and slow memory tiers. As data temperature changes over time. While exclusive memory tiering avoids data redundancy, it necessitates data movement between memory tiers to optimize the performance of data access, i.e., promoting hot data to the fast tier and demoting cold data to the slow tier. Given that all memory tiers are byte-addressable by the CPU and the performance gap between tiers narrows, it remains to be seen whether exclusive tiering is the optimal strategy considering the cost of data movement.

We evaluate the performance of transparent page placement (TPP) [Maruf_23asplos_tpp], more frequent data migrations would occur in a state-of-the-art and the default tiered memory management in Linux. Figure 1 shows the bandwidth of a micro-benchmark that accesses a configurable working set size (WSS) following a Zipfian distribution in a CXL-based memory tiers. It is especially true when the fast-tier memory is under pressure and incapable of holding all hot data. A high frequency of data migration can cause memory thrashing, hurting overall systemperformance. As depicted in Figure 1, the involvement of TPP  [Maruf_23asplos_tpp] in page migration (i.e., TPP in progress) has a significant, negative impact on the performance of a memory-intensive workload. In comparison, when most of the hot data have been moved to the fast tier(i. tiered memory system. More details of the benchmark and the hardware configurations can be found in Section 4. We compare the performance of TPP while it actively migrates pages between tiers for promotion and demotion (denoted as TPP in progress) and when it has finished page relocation (TPP stable) with that of a baseline that disables page migration (no migration). The baseline does not optimize page placement and directly accesses hot pages from the slow tier. The tiered memory testbed is configured with 16GB fast memory (local DRAM) and 16GB slow memory (remote CXL memory). We vary the WSS to fit in (e.g., 10GB) and exceed (e., TPP stable with the g., 24GB) fast memory capacity. Note that the latter requires continuous page migrations between tiers since hot data spills into slow memory. Additionally, we explore two initial data placement strategies in the benchmark. First, the benchmark pre-allocates 10GB WSS) , the workload achieves higher performance than the case without TPP (i.e., No migration) of data in fast memory to emulate the existing memory usage from other applications. Frequency-opt is an allocation strategy that places pages according to the descending order of their access frequencies (hotness). Thus, the hottest pages are initially placed in fast memory until the WSS spills into slow memory. In contrast, Random employs a random allocation policy and may place cold pages initially in fast memory. However, if the

We have important observations from results in Figure 1. First, page migration in TPP incurs significant degradation in application performance. When WSS fits in fast memory, TPP stable, which has successfully migrated all hot pages to fast memory, achieves more than an order of magnitude higher bandwidth than TPP in progress. Most importantly, no migration is consistently and substantially better than TPP in progress, suggesting that the overhead of page migration outweighs its benefit until the migration is completed. Second, TPP never reaches a stable state and enters memory thrashing when WSS is larger than the capacity of fast memory. Third, page migration is crucial to achieving optimal performance if it is possible to move all hot data is larger than the size of the fast tier (the 24GB WSS case), memory thrashing happens, leading to extremely poor performance (i.e. , TPP stable with the 24GB WSS ). Furthermore, given that CXL-based lower-tier memorycan substantially exceed the size of local DRAM [Sun2023DemystifyingCM], the advantage of storing data exclusively for the sake of efficient memory usage becomes less compelling. Instead, more focus should be placed on reducing the overhead when frequent page migration occurs.

to fast memory and the initial placement is sub-optimal, as evidenced by the wide gap between Insight 1: In tiered memorysystems, adopting an exclusive, NUMA-like approach for data management can be less effective, particularly when the fast-tier memorycannot accommodate all the hot data .TPP stable and no migration in the 10GB WSS and random placement test.

Page management has been widely studied to optimize data placement among tiered memory/storage. An effective page management scheme must tackle two primary challenges: In this section, we delve into the design of page management in Linux and analyze its overhead during page migration. We focus our discussions on 1) transparent and accurate identification of the hotness of pageswith minimal overhead used for page placement; how to effectively track memory accesses and identify hot pages, and 2) efficient migration of pages between fast and slow tierswithout adversely impacting application-level performancethe mechanism to migrate a page between memory tiers.

To reduce such overhead,

Linux adopts a lazy scanning mechanism, which defers the scanning until memory is tight, e.g., when the fast-tier memory is under pressure, and the swapping mechanism is invoked for page demotion (e.g., via kswapd). In addition, the kernel maintains two separate PT scanning mechanism to track hot pages, which lays the foundation for its tiered memory management. Linux maintains two LRU lists for each memory tiera memory node: an active list to store hot pages and an inactive list to store for cold pages. Pages can be moved between these two lists during (lazy) scanning. The kernel uses two bits (i.e.By default, all new pages go to the inactive list and will be promoted to the active list according to two flags, PG_referenced and PG_active) to maintain the frequency of each page– a cold page, after two accesses (detected by two scans), will be moved to the active list , in the per-page struct page. Due to lazy scanning, it takes a long time to detect hot pages – PG_reference is set when the access bit in the corresponding PTE is set upon a PTE check and PG_active is set after PG_reference is set for two consecutive times. A page is promoted to the active list when its PG_active flag is set. For file-backed pages, their accesses are handled by the OS through the file system interface, e.g., read() and write(). Therefore, their two flags are updated each time they are accessed. For anonymous pages, e.g., application memory allocated through malloc, since page accesses are directly handled by the MMU hardware and bypass the OS kernel, the updates to their reference flags and LRU list management are only performed during memory reclamation. Under memory pressure, the swapping daemon kswapd scans the inactive list and the corresponding PTEs to update inactive pages’ flags, and reclaims/swaps out those with PG_reference unset. Additionally, kswapd promotes hot pages (i.e., lack of recency. However, timely detecting hot pages for the slow tier is critical. those with PG_active set) to the active list. This lazy scanning mechanism delays access tracking until it is necessary to reduce the tracking overhead, but undermines tracking accuracy.

TPP [Maruf_23asplos_tpp] tackles this problem by tracking page accesses of the slow tier using hint page faults. However, we observed that TPP ’s synchronous page migration approach unexpectedly amplifies its page-fault overhead, leverages Linux’s PT scanning to track hot pages and employs page fault-based tracking to decide whether to promote pages from slow memory. Specifically, TPP sets all pages residing in slow memory (e.g., up to 15 page faults were involved in migrating a single detected hot page from the slow to the fast tiers. This is manifested in Figure 2 – a significant portion of time is consumed by “non-migration page faults” (more details in 3.1) . CXL memory) as inaccessible, and any user access to these pages will trigger a minor page fault, during which TPP decides whether to promote the faulting page. If the faulting page is on the active list, it is migrated (promoted) to the fast tier. Page demotion occurs when fast memory is under pressure and kswapd migrates pages from the inactive list to slow memory.

Accurate and lightweight memory access tracking can be achieved with hardware support, e.g., by adding a PTE count field in hardware that records the number of memory accesses(by hardware) for the associated page [7056027]. However, the proposed approach could be too invasive for hardware-based tracking can increase the complexity and require extensive hardware changes in mainstream architectures (e.g., x86). In practice, the hardware-assisted samplingapproach, , such as via Processor Event-Based Sampling (PEBS) [memtis, Padmapriya_23asplos] on Intel platforms, has been recently proposed to obtain employed to record page access (virtual address) information from sampled hardware events (e.g., LLC misses or store instructions). However, PEBS-based profiling also requires a careful balance between the frequency of sampling and the accuracy of profiling. We observed that the PEBS-based approach [memtis], with a sampling rate incurring low CPU optimized for minimizing overhead, remains coarse-grained and fails to capture many hot pages. Further, the sampling-based approach may not obtain recency of memory access (with a low sampling rate)accurately measure access recency, thus limiting the its ability to make timely migration decisions. Last, PEBS is not accessible across CPU vendors [Padmapriya_23asplos], making the PEBS-based profiling not generic.

Insight 2: A practical, lightweight memory tracking mechanism that can capture both frequency and recency information of page accesses is highly desired.

Typically, scanning-based tracking supports asynchronous, batch-based page promotion /demotion, which redistributes pages among fast/slow tiers based on the periodic updates of data temperature [memtis]. While batching multiple pages for asynchronousmigration could reduce migrationoverhead, it does not favor temporal locality. To migrate . Synchronous migration, e.g., page promotion in TPP, is on-demand pages more promptly, page-fault-based, synchronouspage migration is often used in both Linux and recent research projects [Maruf_23asplos_tpp, Yan_19asplos_nimble]: When a “tagged” page is touched for the first time, page migration is synchronously conducted during handling the hint page fault.Despite being timely, synchronous migration is very costly. First, it resides in the triggered by user access to a page and on the critical path of program execution. During migration, the user program is blocked until migration is completed. Asynchronous migration, e.g., page demotion in TPP, is handled by a kernel thread (i.e., critical pathkswapdof memory access, slowing down the execution of the process requesting such memory access. Further, due to locking (in ), others trying to access the same page must enter a oftentimes off programs’ critical path, when certain criteria are met. Synchronous migration is costly not only because pages are inaccessible during migration but also may involve a large number of page faults.

Figure 2 shows the run time breakdown of the aforementioned benchmark while TPP is actively relocating pages between the two memory tiers. Since page promotion is synchronous, page fault handling and page content copying (i.e., promotion) are executed on the same CPU as the application thread. Page demotion is done through busy-waitkswapd state until the page becomes available remapped).Last, page migration may fail, e.g., due to insufficient space on the destination memory tier, and the need to synchronously re-try further worsens this situation.

This exactly explains the extremely poor performance caused by TPP, as and uses a different core. As shown in Figure 1, i.e. , the cases in “TPP in progress” and “TPP stable” with the 24 GB WSS. The breakdown of time consumption, in Figure 2, further shows that almost 30% of the execution time was spent in the synchronous page promotion (and related) operations, significantly slowing down the workload. synchronous promotion together with page fault handling incurs significant overhead on the application core. In contrast, the demotion core remains largely idle and does not present a bottleneck. As will be discussed in Section 3.1, userspace run time can also be prolonged due to repeated minor page faults (as many as 15) to successfully promote one page. This overhead analysis explains the poor performance of TPP observed in Figure 1.

Insight 3: How can we achieve prompt pagemigration without delaying/pausing the execution of applications?

A long line of pioneering work has explored a wide range of of tiered storage/memory systems, built upon SSDs and HDDs [266253, 8714805, chen2011hystor, 5749736, 10.5555/1855511.1855519, 6557143, 5581609, 6005391, 267006], DRAM and disks [270689, 270754, 269067, papagiannis2020optimizing], HBM and DRAMs [7056027, batman, 6493624], NUMA memory [autoNUMA, damon], PM and DRAM [288741, abulila2019flatflash, Adnan_22hpca, thermostat], local and far memory [Bergman_22ismm, 180194, 180727, 7266934, ruan2020aifm], DRAM and CXL memory [li2023pond, Maruf_23asplos_tpp, memtis], and multiple tiers [wu2021storage, Jonghyeon_21fast, Yan_19asplos_nimble, 288741, kwon2017strata]. We focus on tiered memory systems consisting of DRAM and emerging CXL memory , though the proposed approaches in the emerging byte-addressable memory devices, e.g., CXL memory and PM. Nomad also apply applies to other tiered memory systems such as HBM/DRAM and DRAM/PM. We discuss the most related works as follows.

Inspired by existing lightweight tracking systems, such as Linux’s active and inactive lists and hint page faults, Nomad advances them by incorporating more recency information with no additional CPU overhead. Unlike hardware-assisted approaches [memtis, Padmapriya_23asplos, Loh2012ChallengesIH], Nomad does not require any additional hardware support.

The motivation to develop TPM is to make page migration entirely asynchronous and decoupled from users’ access to the page. As discussed in Section 2.2, the current page migration in Linux is synchronous and on the critical path of users’ data access. For example, the default tiered memory management in Linux, TPP, attempts to migrate a page from the slow tier whenever a user program accesses the page. Since the page is in “protected” mode, the access triggers a minor page fault, leading TPP to attempt the migration. The user program is blocked and makes no progress until the minor page fault is handled and the page is remapped to the fast tier, which can be a time-consuming process. Worse, if the migration fails, the OS remains in function migrate_pages and retries the aforementioned migration until it is successful or reaching a maximum of 10 attempts.

TPM decouples page migration from the critical path of user programs by making the migrating page accessible during the migration process. Therefore, users will access the migrating page from the slow tier before the migration is complete. While accessing a hot page from the slow tier may lead to sub-optimal memory performance, it avoids blocking user access due to the migration, thereby leading to superior user-perceived performance. Figure 3 shows the workflow of TPM. Before migration commences, TPM clears the protection bit of the page frame and adds the page to a migration pending queue. Since the page is no longer protected and it is not yet unmapped from the page table, following accesses to the page will not trigger additional page faults.

TPM starts a migration transaction by clearing the dirty bit of the page (step ❶) and checks the dirty bit after the page is copied to the fast tier to determine whether the transaction was successful. After changing the dirty bit in PTE, TPM issues a TLB shootdown to all cores that ever accessed this page (step ❷). This is to ensure that subsequent writes to the page can be recorded on the PTE. After the TLB shootdown is completed, TPM starts copying the page from the slow tier to the fast tier (step ❸). To commit the transaction, TPM checks the dirty bit by loading the entire PTE using atomic instruction get_and_clear (step ❹). Clearing the PTE is equivalent to unmapping the page and thus another TLB shootdown is needed (step ❺). Note that after unmapping the page from PTE, it becomes inaccessible by users. TPM checks whether the page was dirtied during the page copy (step ❻) and either commits the transaction by remapping the page to the fast tier if the page is clean (step ❼) or otherwise aborts the transaction (step ❽). If the migration is aborted, the original PTE is restored and waits for the next time when TPM is rescheduled to retry the migration. The duration in which the page is inaccessible is between ❹ and ❼/ ❽, significantly shorter than that in TPP (possibly multiple attempts between  ❶ and ❼).

Page migration is a complex procedure that involves memory tracing and updates to the page table for page remapping. The state-of-the-art page fault-based migration approaches, e.g., TPP in Linux [Maruf_23asplos_tpp], employs synchronous page migration, a mechanism in the Linux kernel for moving pages between NUMA nodes. In addition to the extended migration time affecting the critical path of user programs, this mechanism causes excessive page faults when integrated with the existing LRU-based memory tracing. TPP makes per-page migration decisions based on whether the page is on the active LRU list. Nevertheless, in Linux, memory tracing adds pages from the inactive to the active LRU list in batches of 15 requests ²²2The 15 requests could be repeated requests for promoting the same page to the active LRU list, aiming to minimize the queue management overhead. Due to synchronous page migration, TPP may submit multiple requests (up to 15 if the request queue was empty) for a page to be promoted to the active LRU list to initiate the migration process. In the worst case, migrating one page may generate as many as 15 minor page faults.

TPM provides a mechanism to enable asynchronous page migration but requires additional effort to interface with memory tracing in Linux to minimize the number of page faults needed for page migration. As shown in Figure 4, in addition to the inactive and active LRU lists in memory tracing, TPM maintains a separate promotion candidate queue (PCQ) for pages that 1) have been tried for migration but 2) not yet promoted to the active LRU list. Upon each time a minor (hint) page fault occurs and the faulting page is added to PCQ, TPM checks if there are any hot pages in PCQ that have both the active and accessed bits set. These hot pages are then inserted to a migration pending queue, from where they will be tried for asynchronous, transactional migration by a background kernel thread kpromote. Note that TPM does not change how Linux determines the temperature of a page. For example, in Linux, all pages in the active LRU list, which are eligible for migration, have the two memory tracing bits set. However, not all pages with these bits set are in the active list due to LRU list management. TPM bypasses the LRU list management and provides a more efficient method to initiate page migration. If all transactional migrations were successful, TPM guarantees that only one page fault is needed per migration in the presence of LRU list management.

To enable non-exclusive memory tiering, Nomad introduces a one-way page shadowing mechanism to enable a subset of pages resident in the performance tier to have a shadow copy in the capacity tier. Only pages promoted from the slow tier have shadow copies in the slow tier. Shadow copies are the original pages residing on the slow tier before they are unmapped in the page table and migrated to the fast tier. Shadow pages play a crucial role in minimizing the overhead of page migration during periods of memory pressure. Instead of swapping hot and cold pages between memory tiers, page shadowing enables efficient page demotion through page table remapping. This would eliminate half of the page migration overhead, i.e., page demotion, during memory thrashing.

However, tracing updates to the master page poses a significant challenge. Page management in Linux relies heavily on the read-write permission of a page to perform various operations on the page, such as copy-on-write (CoW). While setting master pages as read-only effectively captures all writes, it may affect how these master pages are managed in the kernel. To address this issue, Nomad introduces a procedure called shadow page fault. It still designates all master pages as read-only but preserves the original read-write permission in an unused software bit on the page’s PTE (as shown in Figure 5). We refer to this software bit as shadow r/w. Upon a write to a master page, a page fault occurs. Unlike an ordinary page fault that handles write violation, the shadow page fault, which is invoked if the page’s shadow flag is set in its struct page, restores the read-write permission of the faulting page according to the shadow r/w bit and discards/frees the shadow page on the slow tier. The write may proceed once the shadow page fault returns and reinstates the page to be writable. For read-only pages, tracing shadow pages does not impose additional overhead; for writable pages, it requires one additional shadow page fault to restore their write permission.

Nomad relies on two rounds of TLB shootdown to effectively track updates to a migrating page during transactional page migration. When a page is used by multiple processes or mapped by multiple page tables, its migration involves multiple TLB shootdowns, per each mapping, that need to happen simultaneously. The overhead of handling multiple IPIs could outweigh the benefit of asynchronous page copy. Hence, Nomad deactivates transactional page migration for multi-mapped pages and resorts to the default synchronous page migration mechanism in Linux. As high-latency TLB shootdowns based on IPIs continue to be a performance concern, modern processors, such as ARM, future AMD, and Intel x86 processors, are equipped with ISA extensions for faster broadcast-based [arm_tlb_broadcast, amd_tlb_broadcast] or micro-coded RPC-like [intelcpurpc] TLB shootdowns. These emerging lightweight TLB shootdown methods will greatly reduce the overhead of TLB coherence in tiered memory systems with expanded memory capacity. Nomad will also benefit from the emerging hardware and can be extended to scenarios where more intensive TLB shootdowns are necessary.

We first used micro-benchmarks to investigate the performance of Nomad’s transactional page migration and page shadowing mechanisms. To do this, we developed a micro-benchmark to precisely assess Nomad in a controlled manner. This micro-benchmark involves 1) allocating data to specific segments of the tiered memory, 2) running tests with various working set sizes (WSS) and resident set sizes (RSS); and 3) generating memory accesses to the WSS data that mimic real-world memory access patterns with a Zipfian distribution. We created three distinct scenarios , which represent the representing small, medium, and large WSS , for the micro-benchmarking. We conducted all the micro-benchmarking on Platform B – a tiered memory system with 16 GB local DRAM and 16 GB remote CXL memory. all platforms. Note that Platform B produced similar results as A (hence not listed).

Figure LABEL:fig:smallwss illustrates Figures 7 (a) (Platform A), 8 (a) (Platform C), and 9 (a) (Platform D) show that in the initial phase, where intensive during which page migration was conducted intensively (i.e., migration in progress), both Nomad and Memtis demonstrated equivalent similar performance regarding memory bandwidth for readsand writes . During this period, they outperformed TSS significantly , with improvements ranging from 100% to 300%. It indicates that although page-fault-based page migration in Nomad could incur more overhead than the PEBS-based approach in Memtis, a timely page migration (when the WSS is small) could offset this disadvantage. In contrast, for writes (especially on Platform A), Nomad achieved lower bandwidth than Memtis due to additional overhead to support page shadowing (Section 3.2). That is, Nomad sets the master page (migrated to the fast tier) as read-only, and the first write to this page causes a page fault, hence detrimental to write-intensive workloads. The observation also applies to other write-intensive test cases, as will be discussed shortly. On the other hand, Nomad outperformed TPP significantly on Platform A and D and achieved similar performance on Platform C. The performance benefit came stemmed from the asynchronous , transactional page migration for Nomad, which does not prevent workloads from accessing pages during migration (Section 3.1)and asynchronous, background page migration for Memtis, which merely blocked the execution of the micro-benchmark. .

Figure LABEL:fig:smallwss further shows the results in In the stable phase (i.e., migration stable), where most of the WSS data was had been migrated from the CXL memory or PM to the local DRAM. Both , both Nomad and TPP achieved equivalent similar read/write performance bandwidth. This was because memory accesses were all primarily served by the local DRAMand there were few or no page migrations , as listed , with few page migrations occurring, as shown in Table LABEL:tab:pagemigrationnumb. Surprisingly, the performance of Memtis was the worst(less than 2. Memtis performed the worst, achieving as low as 40% of the other two). In fact, the performance with Memtisdid not change much performance of the other two approaches. We further observed that, in most cases, Memtis’s performance increased only slightly from the initial phase to the stable phase. Possible reasons include Furthermore, with a higher rate of cooling down hot pages, Memtis-QuickCool performed better than Memtis-Default. Possible reasons for Memtis’s low performance include: 1) Memtis’’s PEBS-based sampling was too coarse-grained and failed to capture most hot pages on the CXL memory ; (i.e., as stated earlier, LLC misses to CXL memory cannot be captured by Memtis); and 2) Memtis’’s background migration thread could not timely migrate pages from the local DRAM to the CXL memory . We only observed promptly. We observed only a small number of page migrations occurring conducted by Memtis during both the initial and stable phases(not , as listed in Table LABEL:tab:pagemigrationnumb). Hence, in 2. Consequently, in the stable phase, many memory accesses were still served by the slow CXL memory , the slow memory tier still handled many memory accesses, leading to low performance.

Similar to Unlike the small WSS case, Figure LABEL:fig:mediumwss demonstrates Figures 7 (b), 8 (b), and 9 (b) show that during the initial phase, where heavy page migrations were conducted (migration in progress), both Nomad and Memtis achieved comparable memory bandwidths TPP generally achieved lower memory bandwidth for read and write operations . In this period, they significantly outperformed TPP, showing improvements in the range of 2x to 3x. Compared with the small WSScase, in the medium compared to Memtis. This is because, under the medium WSS, the system experienced higher memory pressure than in the small WSS case, causing Nomad and TPP to conduct more page migrations were conducted in the initial phase – e.g., more than (2x as listed - 6x) and incur higher overhead than Memtis, as shown in Table  LABEL:tab:pagemigrationnumb. This is because a larger portion of the WSS data was initially allocated in the CXL memory 2. Conversely, Memtis performed fewer page migrations than the small WSS case(11 GB vs. 4 GB).

Figure LABEL:fig:mediumwss presents the findings from the stable phase (migration stable). In this phase, Nomad and Memtis exhibited nearly the same performance in terms of write operations. However, for read operations, Nomad notably excelled, outperforming the other two approaches substantially – for instance, it achieved 4x the performance of TSS and 2x the performance of Memtis. In . One possible reason is that: It did not capture the LLC misses of CXL memory and mostly relied on TLB misses for tracking memory access, but the events of TLB misses were overshadowed by extensive local LLC miss events in the medium WSS case. The less effective PEBS-based approach in Memtis, resulting in fewer page migrations, we observed that, during the stable phase, a significant number of page migrationswere still triggered, as listed in Table LABEL:tab:pagemigrationnumb. The reason is that although the WSS of 13.5 GB is smaller than the local DRAM capacity of 16 GB, the combined size of the WSS data and the system’s data exceeds the threshold for space reclaiming, triggering the demotion thread to begin migrating pages to the CXL memory. The demoted pages belonging to the WSS were promptly promoted back to the local DRAM shortly, leading to memory thrashing. redemptively maintained the system’s performance under high memory pressure.

As the system entered the stable phase, Nomad can significantly mitigate the performance degradation for read operations (resulting from memory thrashing) due to its achieved higher performance for both reads and writes than the initial phase due to 1) most hot pages residing in the fast tier and thus 2) fewer page migrations being triggered. Compared to Memtis-Default, Nomad achieved higher read performance and similar write performance. The reason that Nomad was particularly superior in read performance – the read bandwidth was the highest among all test cases during the stable state across all three platforms – was due to Nomad’s non-exclusive memory tiering. Recall that Nomad’s page shadowing mechanism enables a subset of pages resident in the local DRAM to have a shadow copy in the CXL memory or PM (Section 3.2). This mechanism benefits the read operations when thrashing occurs (e.g., with a medium WSS) – when a page is selected for demotion and meanwhile Nomad retains its copy in the CXL memory or PM, the page is discarded rather than migrated. Unfortunately for writes, as the local version has been modified during demotion, the new version needs to be migrated to the CXL memory. But However, as discussed earlier, the page shadowing mechanism could be detrimental to write operations due to additional page faults. This explains why Nomad still managed to deliver the best performance among the three approaches. achieved similar write performance as Memtis-Default and sometimes worse than Memtis-QuickCool. Finally, Nomad again significantly outperformed TPP in all cases.

Figure LABEL:fig:largewss presents

Figures 7 (c), 8 (c), and 9 (c) present the performance in both the initial phase and the the stable phase. InterestinglyAs expected, both Nomad and TPP, tending which tend to promptly migrate detected hot data from the CXL memory ， or PM, performed worse than Memtis, which only conducted background promotion/demotion. This is because , the severe memory thrashing caused by the large WSS made Nomad and TPP , frequently migrated frequently migrate pages between the local and remote memory, greatly degrading the micro-benchamrk’micro-benchmark’s performance. In contrast, much far fewer page migrations occurred with Memtis. HoweverAdditionally, Nomad consistently outperformed TPP by a range of 2x - 5x, again due to its asynchronous, transactional page migration. In additionAgain, Nomad’s page shadowing ’s page shadowing helped achieve higher performanceread performance, especially during the stable state on Platform A. Regardless, frequent page migrations become counterproductive when the WSS significantly exceeds the capacity of fast memory.

Performance comparisons with PageRank between non-migration, TPP, Memtis, and Nomad.

Performance comparisons with Liblinear between non-migration, TPP, Memtis, and Nomad.

We continued the evaluation of Nomad using three representative real-world applications with unique memory access patterns: Redis [redis], PageRank [pagerankwiki], and Liblinear [liblinear]. We ran these three applications on top of three four different platforms (Table 1) to reveal the benefits and potential limitations of current page management systems for emerging tiered with two cases: 1) a small RSS (under 32 GB) working with all platforms and 2) a large RSS (over 32 GB) that only Platform C and D can run due to larger PM or CXL memory.

unless otherwise specified. Case 1: We set recordcount recordcount to 6 million and operationcount operationcount to 8 million. After pre-loading the dataset into the database, we used a customized tool to demote all Redis’ memory pages to the slow memory tier . Then, we started to run tier before starting the experiment. The RSS of this case was 13GB.

Case 2: We increased the RSS by setting recordcount recordcount to 10 million and operationcount operationcount to 12 million. We used the same way to demote demoted all the memory pages to the slow tier in the same way. The RSS of this case was 24GB.

Case 3: We increased the total operations while keeping the same RSS by setting recordcount to 10 million and operationcount to 12 million. After kept the same total operations and RSS as Case 2. However, after pre-loading the dataset, we did not not demote any memory pages. The RSS was 24GB.

Figure LABEL:ycsb shows that , compared to TPP, Consistent with the micro-benchmarking results, Figure 11 shows that Nomad consistently delivered superior performance for the latency-sensitive key-value store across all tested platforms in the three cases. The enhanced performance attributable to (in terms of operations per second) compared to TPP across all platforms in all cases. In addition, Nomad stems from its asynchronous, transactional page migration and page shadowing approaches , which merely block the execution of applications. Notably,outperformed Memtis when the WSS was small (i.e., for Case 1), but suffered more performance degradation as the WSS increased (i.e., for Case 2 and 3) due to an increased number of page migrations and additional overhead. Finally, all the page migration approaches underperformed compared to the “no migration” scenario. It is because the memory accesses generated by the YCSB workload were mostly “random,” rendering migrating pages to the fast tier less effective, as those pages were unlikely to be accessed again during the test. It indicates once again that page migration could incur nontrivial overhead, and a strategy to smartly switch it on/off is needed.

We further increased the RSS of the database and operations of YCSB by setting the recordcount to 20 million and operationcount to 30 million. The RSS for this case was 36.5GB, exceeding the total size of the tiered memory on Platforms A and B, thus only being able to run on Platforms C and D. We tested two initial memory placement strategies for the database – Thrashed (demoting all memory pages to the slow tier) and Normal (placing the memory pages starting from local DRAM). In this larger-scale case with more severe memory thrashing, as shown in Figure 16, Nomad demonstrated the highest performance in case 3. This is due to that a smaller amount of data (8 GB) was initially allocated onthe slower tier in case 3, compared to the larger initial allocations (13 GB and 16 GB)in the other two cases. led to lower performance compared to Memtis due to massive page migration operations but still outperformed TPP.

Figure LABEL:pagerank 12 illustrates that there is was negligible variance in performance between scenarios with page migrations (using Nomad and TPP) and without page migrations (no migration). The results suggest that: 1) For non-latency-sensitive applications, such as PageRank, using CXL memory (as in Platform A) can significantly expand the local DRAM capacity without adversely impacting application-level performance. 2) In such scenarios, page migration appears to be unnecessary. In addition, the application-level performance diminishes with PM (as observed in Platform C), partly due to PM’s higher latency compared to the CXL memory. These findings also reveal that the overhead associated with Nomad’s page migration minimally influences PageRank’s performance. Additionally, it is was observed that among all evaluated scenarios, Memtis exhibits exhibited the least efficient performance.

Figure 16 shows the case when we scaled the RSS to a very large scale on Platform C & D. When the PageRank program started, it first used up to 100GB memory, then its RSS size dropped to 45GB to 50GB. Nomad achieved 2x the performance of TPP (both platforms) and slightly better than Memtis (Plactform C), due to more frequent page migrations – the local DRAM (16 GB) was not large enough to accommodate the WSS in this case.

Figure LABEL:liblinear illustrates 13 demonstrate that both Nomad and TPP significantly outperformed the “no migration” scenario on Platform B, which utilizes CXL-based memory tiering. However, this performance advantage was not evident on Platform C, featuring PM-based memory tiering, where all approaches exhibited comparable performance. Further analysis revealed a sharp difference in page promotions between the platforms: 12 million on Platform B compared to just 1 million on Platform C. This discrepancy is attributable to the differing speeds of CXL memory and PM. This observation indicates that the efficacy of page migration is closely linked to the performance characteristics of the underlying memory media. and Memtis across all platforms, with performance improvement ranging from 20% to 150%. This result further illustrates that when the WSS is smaller than the local DRAM, Nomad and TPP can substantially enhance application performance by timely migrating hot pages to the faster memory tier. Figure 16 shows that with a much larger model and RSS when running Liblinear, NOMAD consistently achieved high performance across all cases. In contrast, TPP’s performance significantly declined, likely due to inefficiency issues, as frequent, high bursts in kernel CPU time were observed during TPP execution.

Our observations from Figure LABEL:pagerank and Figure LABEL:liblinear also reveal that the