This paper summarizes the configuration, benchmark steps, and results of a system that was built ... more This paper summarizes the configuration, benchmark steps, and results of a system that was built to improve upon the 1TB JouleSort benchmark [8] (Daytona Category). We use a desktop machine equipped with energyefficient SSDs and show that we can improve upon the current 2013 [4] and 2019 [6] JouleSort world records by 17% and 15%, respectively. Our desktop machine is a modified Asus ROG Strix G15CK build [2]— equipped with Intel Core i7-10700 CPU running @ 2.9 GHz, and paired with 16 GB DDR4 DRAM. We use 2× 1TB SK hynix Gold P31 NVMe SSDs and a 2TB Samsung 980 Pro NVMe SSD. Using this hardware, we are able to perform a 1TB sort (with 10 records) in an average time of 1379 seconds running at 100.4 W—requiring a total of 138,416 joules.
It is becoming increasingly popular for distributed systems to exploit offload to reduce load on ... more It is becoming increasingly popular for distributed systems to exploit offload to reduce load on the CPU. Remote Direct Memory Access (RDMA) offload, in particular, has become popular. However, RDMA still requires CPU intervention for complex offloads that go beyond simple remote memory access. As such, the offload potential is limited and RDMAbased systems usually have to work around such limitations. We present RedN, a principled, practical approach to implementing complex RDMA offloads, without requiring any hardware modifications. Using self-modifying RDMA chains, we lift the existing RDMA verbs interface to a Turing complete set of programming abstractions. We explore what is possible in terms of offload complexity and performance with a commodity RDMA NIC. We show how to integrate these RDMA chains into applications, such as the Memcached keyvalue store, allowing us to offload complex tasks such as key lookups. RedN can reduce the latency of key-value get operations by up to 2.6× compared to state-of-the-art KV designs that use one-sided RDMA primitives (e.g., FaRM-KV), as well as traditional RPC-over-RDMA approaches. Moreover, compared to these baselines, RedN provides performance isolation and, in the presence of contention, can reduce latency by up to 35× while providing applications with failure resiliency to OS and process crashes.
A common pattern in the architectures of modern interac-tive web-services is that of large reques... more A common pattern in the architectures of modern interac-tive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensi-tive to the tail of latency distribution of the backend tier. The large number of decentralized application servers and skewed workload patterns exacerbate the challenge in ad-dressing this problem. We address these challenges through BetteR Batch (BRB). By carefully scheduling requests in a decentralized and task-aware manner, BRB enables low-latency distributed storage systems to deliver predictable performance in the presence of large request fan-outs. Our preliminary simulation results based on production workloads show that our proposed de-sign is at the 99th percen...
Disaggregated, or non-local, file storage has become a common design pattern in cloud systems, of... more Disaggregated, or non-local, file storage has become a common design pattern in cloud systems, offering benefits of resource pooling and server specialization, where the inherent overhead of separating compute and storage is mostly hidden by storage device latency. We take an alternate approach, motivated by the commercial availability of very low latency non-volatile memory (NVM). By colocating computation and NVM storage, we can provide applications much higher I/O performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated cache coherence protocol for managing a set of colocated NVM storage devices as a layer. Unlike disaggregated file stores, Assise avoids the read and write amplification of page granularity operations. Instead, remote NVM serves as an intermediate, byte-addressable cache between colocated NVM and slower storage, such as SSDs. We compare Assise to Ceph/B...
A common pattern in the architectures of modern interactive web-services is that of large request... more A common pattern in the architectures of modern interactive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensitive to the tail of latency distribution of the backend tier. To address such challenges, we present BetteR Batch (BRB), a system that carefully schedules requests in a decentralized and task-aware manner. In doing so, we improve performance predictability of data stores in the presence of large request fan-outs. We show via simulations that our proposed design improves latencies over the state-of-the-art by a factor of 2.
The adoption of low latency persistent memory modules (PMMs) upends the long-established model of... more The adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, P...
Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, 2015
A common pattern in the architectures of modern interactive web-services is that of large request... more A common pattern in the architectures of modern interactive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensitive to the tail of latency distribution of the backend tier. The large number of decentralized application servers and skewed workload patterns exacerbate the challenge in addressing this problem. We address these challenges through BetteR Batch (BRB). By carefully scheduling requests in a decentralized and task-aware manner, BRB enables low-latency distributed storage systems to deliver predictable performance in the presence of large request fan-outs. Our preliminary simulation results based on production workloads show that our proposed design is at the 99th percentile...
This report provides an extended evaluation of Kurma, a practical implementation of a geo-distrib... more This report provides an extended evaluation of Kurma, a practical implementation of a geo-distributed load balancer for backend storage systems. In this report we demonstrate the ability of distrib ...
Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles CD-ROM, 2021
In multi-tenant systems, the CPU overhead of distributed file systems (DFSes) is increasingly a b... more In multi-tenant systems, the CPU overhead of distributed file systems (DFSes) is increasingly a burden to application performance. CPU and memory interference cause degraded and unstable application and storage performance, in particular for operation latency. Recent client-local DFSes for persistent memory (PM) accelerate this trend. DFS offload to SmartNICs is a promising solution to these problems, but it is challenging to fit the complex demands of a DFS onto simple SmartNIC processors located across PCIe. We present LineFS, a SmartNIC-offloaded, high-performance DFS with support for client-local PM. To fully leverage the SmartNIC architecture, we decompose DFS operations into execution stages that can be offloaded to a parallel datapath execution pipeline on the SmartNIC. LineFS offloads CPU-intensive DFS tasks, like replication, compression, data publication, index and consistency management to a Smart-NIC. We implement LineFS on the Mellanox BlueField Smart-NIC and compare it to Assise, a state-of-the-art PM DFS. LineFS improves latency in LevelDB up to 80% and throughput in Filebench up to 79%, while providing extended DFS availability during host system failures. * Work done while at KAIST.
Proceedings of the ACM Symposium on Cloud Computing, 2018
The increasing density of globally distributed datacenters reduces the network latency between ne... more The increasing density of globally distributed datacenters reduces the network latency between neighboring datacenters and allows replicated services deployed across neighboring locations to share workload when necessary, without violating strict Service Level Objectives (SLOs). We present Kurma, a practical implementation of a fast and accurate load balancer for geo-distributed storage systems. At run-time, Kurma integrates network latency and service time distributions to accurately estimate the rate of SLO violations for requests redirected across geo-distributed datacenters. Using these estimates, Kurma solves a decentralized rate-based performance model enabling fast load balancing (in the order of seconds) while taming global SLO violations. We integrate Kurma with Cassandra, a popular storage system. Using real-world traces along with a geo-distributed deployment across Amazon EC2, we demonstrate Kurma's ability to effectively share load among datacenters while reducing SLO violations by up to a factor of 3 in high load settings or reducing the cost of running the service by up to 17%.
Proceedings of the Twelfth European Conference on Computer Systems, 2017
We tackle the problem of reducing tail latencies in distributed key-value stores, such as the pop... more We tackle the problem of reducing tail latencies in distributed key-value stores, such as the popular Cassandra database. We focus on workloads of multiget requests, which batch together access to several data elements and parallelize read operations across the data store machines. We first analyze a production trace of a real system and quantify the skew due to multiget sizes, key popularity, and other factors. We then proceed to identify opportunities for reduction of tail latencies by recognizing the composition of aggregate requests and by carefully scheduling bottleneck operations that can otherwise create excessive queues. We design and implement a system called Rein, which reduces latency via inter-multiget scheduling using low overhead techniques. We extensively evaluate Rein via experiments in Amazon Web Services (AWS) and simulations. Our scheduling algorithms reduce the median, 95 th , and 99 th percentile latencies by factors of 1.5, 1.5, and 1.9, respectively.
A commonly held belief is that traffic engineering and routing changes are infrequent. However, b... more A commonly held belief is that traffic engineering and routing changes are infrequent. However, based on our measurements over a number of years of traffic between data centers in one of the largest cloud provider's networks, we found that it is common for flows to change paths at ten-second intervals or even faster. These frequent path and, consequently, latency variations can negatively impact the performance of cloud applications, specifically, latency-sensitive and geo-distributed applications. Our recent measurements and analysis focused on observing path changes and latency variations between different Amazon aws regions. To this end, we devised a path change detector that we validated using both ad hoc experiments and feedback from cloud networking experts. The results provide three main insights: (1) Traffic Engineering (TE) frequently moves (TCP and UDP) flows among network paths of different latency, (2) Flows experience unfair performance, where a subset of flows betw...
This paper summarizes the configuration, benchmark steps, and results of a system that was built ... more This paper summarizes the configuration, benchmark steps, and results of a system that was built to improve upon the 1TB JouleSort benchmark [8] (Daytona Category). We use a desktop machine equipped with energyefficient SSDs and show that we can improve upon the current 2013 [4] and 2019 [6] JouleSort world records by 17% and 15%, respectively. Our desktop machine is a modified Asus ROG Strix G15CK build [2]— equipped with Intel Core i7-10700 CPU running @ 2.9 GHz, and paired with 16 GB DDR4 DRAM. We use 2× 1TB SK hynix Gold P31 NVMe SSDs and a 2TB Samsung 980 Pro NVMe SSD. Using this hardware, we are able to perform a 1TB sort (with 10 records) in an average time of 1379 seconds running at 100.4 W—requiring a total of 138,416 joules.
It is becoming increasingly popular for distributed systems to exploit offload to reduce load on ... more It is becoming increasingly popular for distributed systems to exploit offload to reduce load on the CPU. Remote Direct Memory Access (RDMA) offload, in particular, has become popular. However, RDMA still requires CPU intervention for complex offloads that go beyond simple remote memory access. As such, the offload potential is limited and RDMAbased systems usually have to work around such limitations. We present RedN, a principled, practical approach to implementing complex RDMA offloads, without requiring any hardware modifications. Using self-modifying RDMA chains, we lift the existing RDMA verbs interface to a Turing complete set of programming abstractions. We explore what is possible in terms of offload complexity and performance with a commodity RDMA NIC. We show how to integrate these RDMA chains into applications, such as the Memcached keyvalue store, allowing us to offload complex tasks such as key lookups. RedN can reduce the latency of key-value get operations by up to 2.6× compared to state-of-the-art KV designs that use one-sided RDMA primitives (e.g., FaRM-KV), as well as traditional RPC-over-RDMA approaches. Moreover, compared to these baselines, RedN provides performance isolation and, in the presence of contention, can reduce latency by up to 35× while providing applications with failure resiliency to OS and process crashes.
A common pattern in the architectures of modern interac-tive web-services is that of large reques... more A common pattern in the architectures of modern interac-tive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensi-tive to the tail of latency distribution of the backend tier. The large number of decentralized application servers and skewed workload patterns exacerbate the challenge in ad-dressing this problem. We address these challenges through BetteR Batch (BRB). By carefully scheduling requests in a decentralized and task-aware manner, BRB enables low-latency distributed storage systems to deliver predictable performance in the presence of large request fan-outs. Our preliminary simulation results based on production workloads show that our proposed de-sign is at the 99th percen...
Disaggregated, or non-local, file storage has become a common design pattern in cloud systems, of... more Disaggregated, or non-local, file storage has become a common design pattern in cloud systems, offering benefits of resource pooling and server specialization, where the inherent overhead of separating compute and storage is mostly hidden by storage device latency. We take an alternate approach, motivated by the commercial availability of very low latency non-volatile memory (NVM). By colocating computation and NVM storage, we can provide applications much higher I/O performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated cache coherence protocol for managing a set of colocated NVM storage devices as a layer. Unlike disaggregated file stores, Assise avoids the read and write amplification of page granularity operations. Instead, remote NVM serves as an intermediate, byte-addressable cache between colocated NVM and slower storage, such as SSDs. We compare Assise to Ceph/B...
A common pattern in the architectures of modern interactive web-services is that of large request... more A common pattern in the architectures of modern interactive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensitive to the tail of latency distribution of the backend tier. To address such challenges, we present BetteR Batch (BRB), a system that carefully schedules requests in a decentralized and task-aware manner. In doing so, we improve performance predictability of data stores in the presence of large request fan-outs. We show via simulations that our proposed design improves latencies over the state-of-the-art by a factor of 2.
The adoption of low latency persistent memory modules (PMMs) upends the long-established model of... more The adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, P...
Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, 2015
A common pattern in the architectures of modern interactive web-services is that of large request... more A common pattern in the architectures of modern interactive web-services is that of large request fan-outs, where even a single end-user request (task) arriving at an application server triggers tens to thousands of data accesses (sub-tasks) to different stateful backend servers. The overall response time of each task is bottlenecked by the completion time of the slowest sub-task, making such workloads highly sensitive to the tail of latency distribution of the backend tier. The large number of decentralized application servers and skewed workload patterns exacerbate the challenge in addressing this problem. We address these challenges through BetteR Batch (BRB). By carefully scheduling requests in a decentralized and task-aware manner, BRB enables low-latency distributed storage systems to deliver predictable performance in the presence of large request fan-outs. Our preliminary simulation results based on production workloads show that our proposed design is at the 99th percentile...
This report provides an extended evaluation of Kurma, a practical implementation of a geo-distrib... more This report provides an extended evaluation of Kurma, a practical implementation of a geo-distributed load balancer for backend storage systems. In this report we demonstrate the ability of distrib ...
Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles CD-ROM, 2021
In multi-tenant systems, the CPU overhead of distributed file systems (DFSes) is increasingly a b... more In multi-tenant systems, the CPU overhead of distributed file systems (DFSes) is increasingly a burden to application performance. CPU and memory interference cause degraded and unstable application and storage performance, in particular for operation latency. Recent client-local DFSes for persistent memory (PM) accelerate this trend. DFS offload to SmartNICs is a promising solution to these problems, but it is challenging to fit the complex demands of a DFS onto simple SmartNIC processors located across PCIe. We present LineFS, a SmartNIC-offloaded, high-performance DFS with support for client-local PM. To fully leverage the SmartNIC architecture, we decompose DFS operations into execution stages that can be offloaded to a parallel datapath execution pipeline on the SmartNIC. LineFS offloads CPU-intensive DFS tasks, like replication, compression, data publication, index and consistency management to a Smart-NIC. We implement LineFS on the Mellanox BlueField Smart-NIC and compare it to Assise, a state-of-the-art PM DFS. LineFS improves latency in LevelDB up to 80% and throughput in Filebench up to 79%, while providing extended DFS availability during host system failures. * Work done while at KAIST.
Proceedings of the ACM Symposium on Cloud Computing, 2018
The increasing density of globally distributed datacenters reduces the network latency between ne... more The increasing density of globally distributed datacenters reduces the network latency between neighboring datacenters and allows replicated services deployed across neighboring locations to share workload when necessary, without violating strict Service Level Objectives (SLOs). We present Kurma, a practical implementation of a fast and accurate load balancer for geo-distributed storage systems. At run-time, Kurma integrates network latency and service time distributions to accurately estimate the rate of SLO violations for requests redirected across geo-distributed datacenters. Using these estimates, Kurma solves a decentralized rate-based performance model enabling fast load balancing (in the order of seconds) while taming global SLO violations. We integrate Kurma with Cassandra, a popular storage system. Using real-world traces along with a geo-distributed deployment across Amazon EC2, we demonstrate Kurma's ability to effectively share load among datacenters while reducing SLO violations by up to a factor of 3 in high load settings or reducing the cost of running the service by up to 17%.
Proceedings of the Twelfth European Conference on Computer Systems, 2017
We tackle the problem of reducing tail latencies in distributed key-value stores, such as the pop... more We tackle the problem of reducing tail latencies in distributed key-value stores, such as the popular Cassandra database. We focus on workloads of multiget requests, which batch together access to several data elements and parallelize read operations across the data store machines. We first analyze a production trace of a real system and quantify the skew due to multiget sizes, key popularity, and other factors. We then proceed to identify opportunities for reduction of tail latencies by recognizing the composition of aggregate requests and by carefully scheduling bottleneck operations that can otherwise create excessive queues. We design and implement a system called Rein, which reduces latency via inter-multiget scheduling using low overhead techniques. We extensively evaluate Rein via experiments in Amazon Web Services (AWS) and simulations. Our scheduling algorithms reduce the median, 95 th , and 99 th percentile latencies by factors of 1.5, 1.5, and 1.9, respectively.
A commonly held belief is that traffic engineering and routing changes are infrequent. However, b... more A commonly held belief is that traffic engineering and routing changes are infrequent. However, based on our measurements over a number of years of traffic between data centers in one of the largest cloud provider's networks, we found that it is common for flows to change paths at ten-second intervals or even faster. These frequent path and, consequently, latency variations can negatively impact the performance of cloud applications, specifically, latency-sensitive and geo-distributed applications. Our recent measurements and analysis focused on observing path changes and latency variations between different Amazon aws regions. To this end, we devised a path change detector that we validated using both ad hoc experiments and feedback from cloud networking experts. The results provide three main insights: (1) Traffic Engineering (TE) frequently moves (TCP and UDP) flows among network paths of different latency, (2) Flows experience unfair performance, where a subset of flows betw...
Uploads
Papers by Waleed Reda