SkyRDMA: Fully Offloaded Cloud RDMA Virtualization

Abstract

Remote Direct Memory Access (RDMA) has become the cornerstone of many cloud applications, yet extending bare-metal RDMA performance and reliability to multi-tenant public clouds remains elusive. Existing RDMA virtualization solutions struggle at scale, suffering from performance degradation, weak isolation, and inflexible control. We present SkyRDMA, a new architecture for cloud RDMA virtualization that addresses these limitations holistically. At its core, SkyRDMA introduces a socket-direct RNIC model built via multi-root PCIe, allowing a single RNIC to attach directly to multiple CPU sockets. This design eliminates cross-socket RDMA traffic for tenants without requiring a NIC per CPU socket. On top of this substrate, SkyRDMA provides a reconfigurable, end-to-end QoS framework that jointly enforces isolation at both RNIC ports and PCIe transport, and offloads the RDMA control plane from the host kernel into an embedded SoC on the NIC, minimizing control operation overhead and enabling runtime policy updates without downtime. We implement SkyRDMA on FPGA-based RNICs and deploy it in production cloud RDMA environment. Experiments show that SkyRDMA delivers near bare-metal performance, significantly reduces interference from co-located tenants and services, and enables fast, safe control-plane evolution, all while remaining compatible with existing RDMA applications.