NimbusNet: Building a High‑Performance Echo & Chat Server Across Boost.Asio and Io_uring

We design and benchmark a cross‑platform echo & chat server that scales from laptops to low‑latency Linux boxes. Starting with a Boost.Asio baseline, we add UDP and finally an io_uring implementation that closes the gap with DPDK‑style kernel‑bypass—all while preserving a single, readable codebase. io_uring Full code is available here: https://github.com/hariharanragothaman/nimbus-echo https://github.com/hariharanragothaman/nimbus-echo Motivation Real‑time collaboration tools, multiplayer games, and HFT gateways all live or die by tail latency. Traditional blocking sockets waste cycles on context switches; bespoke bypass stacks (XDP, DPDK) achieve greatness at the cost of portability. tail latency NimbusNet shows you can split the difference: NimbusNet Run anywhere with Boost.Asio (macOS, Windows, CI containers). Drop latency ~2× with UDP by eliminating TCP’s ordering tax. Unlock sub‑25 µs RTT on Linux via io_uring—no kernel patches, no CAP_NET_RAW. Run anywhere with Boost.Asio (macOS, Windows, CI containers). Run anywhere Drop latency ~2× with UDP by eliminating TCP’s ordering tax. Drop latency ~2× Unlock sub‑25 µs RTT on Linux via io_uring—no kernel patches, no CAP_NET_RAW. Unlock sub‑25 µs io_uring Build Environment: Build Environment: Host Toolchain Runtime Variant(s) macOS 14.5 (M2 Pro) Apple clang 15, Homebrew Boost 1.85 Boost.Asio / TCP & UDP Ubuntu 24.04 (x86‑64) GCC 13, liburing 2.6 Boost.Asio / TCP & UDP, io_uring / TCP GitHub Actions macos‑14, ubuntu‑24.04 CI build + tests Host Toolchain Runtime Variant(s) macOS 14.5 (M2 Pro) Apple clang 15, Homebrew Boost 1.85 Boost.Asio / TCP & UDP Ubuntu 24.04 (x86‑64) GCC 13, liburing 2.6 Boost.Asio / TCP & UDP, io_uring / TCP GitHub Actions macos‑14, ubuntu‑24.04 CI build + tests Host Toolchain Runtime Variant(s) Host Host Toolchain Toolchain Runtime Variant(s) Runtime Variant(s) macOS 14.5 (M2 Pro) Apple clang 15, Homebrew Boost 1.85 Boost.Asio / TCP & UDP macOS 14.5 (M2 Pro) macOS 14.5 (M2 Pro) Apple clang 15, Homebrew Boost 1.85 Apple clang 15, Homebrew Boost 1.85 Boost.Asio / TCP & UDP Boost.Asio / TCP & UDP Ubuntu 24.04 (x86‑64) GCC 13, liburing 2.6 Boost.Asio / TCP & UDP, io_uring / TCP Ubuntu 24.04 (x86‑64) Ubuntu 24.04 (x86‑64) GCC 13, liburing 2.6 GCC 13, liburing 2.6 liburing 2.6 Boost.Asio / TCP & UDP, io_uring / TCP Boost.Asio / TCP & UDP, io_uring / TCP io_uring GitHub Actions macos‑14, ubuntu‑24.04 CI build + tests GitHub Actions GitHub Actions macos‑14, ubuntu‑24.04 macos‑14, ubuntu‑24.04 CI build + tests CI build + tests Phase 1 – Establishing the Baseline (Boost.Asio, TCP) Phase 1 – Establishing the Baseline (Boost.Asio, TCP) We begin with a minimal asynchronous echo service that compiles natively on macOS. Boost.Asio’s Proactor‑styleasync_read_some/async_writegives us a platform‑agnostic way to experiment before introducing kernel‑bypass techniques. async_read_some async_write #include #include #include using boost::asio::ip::tcp; class EchoSession : public std::enable_shared_from_this { tcp::socket socket_; std::array buf_{}; public: explicit EchoSession(tcp::socket s) : socket_(std::move(s)) {} void start() { read(); } private: void read() { auto self = shared_from_this(); socket_.async_read_some(boost::asio::buffer(buf_), [this, self](auto ec, std::size_t n) { if (!ec) write(n); }); } void write(std::size_t n) { auto self = shared_from_this(); boost::asio::async_write(socket_, boost::asio::buffer(buf_, n), [this, self](auto ec, std::size_t) { if (!ec) read(); }); } }; int main() { boost::asio::io_context io; tcp::acceptor acc(io, {tcp::v4(), 9000}); std::function do_accept = [&]() { acc.async_accept([&](auto ec, tcp::socket s) { if (!ec) std::make_shared (std::move(s))->start(); do_accept(); }); }; do_accept(); std::cout #include #include using boost::asio::ip::tcp; class EchoSession : public std::enable_shared_from_this { tcp::socket socket_; std::array buf_{}; public: explicit EchoSession(tcp::socket s) : socket_(std::move(s)) {} void start() { read(); } private: void read() { auto self = shared_from_this(); socket_.async_read_some(boost::asio::buffer(buf_), [this, self](auto ec, std::size_t n) { if (!ec) write(n); }); } void write(std::size_t n) { auto self = shared_from_this(); boost::asio::async_write(socket_, boost::asio::buffer(buf_, n), [this, self](auto ec, std::size_t) { if (!ec) read(); }); } }; int main() { boost::asio::io_context io; tcp::acceptor acc(io, {tcp::v4(), 9000}); std::function do_accept = [&]() { acc.async_accept([&](auto ec, tcp::socket s) { if (!ec) std::make_shared (std::move(s))->start(); do_accept(); }); }; do_accept(); std::cout << "⚡ NimbusNet echo listening on 0.0.0.0:9000\n"; io.run(); } 2 – UDP vs. TCP: When Reliability Becomes a Tax TCP’s 3‑way handshake, retransmit queues, and head‑of‑line blocking are lifesavers for file transfers—and millstones for chats that can drop an occasional emoji. TCP bakes in ordering, re‑transmission, and congestion avoidance; these guarantees come at the cost of extra context switches and kernel bookkeeping. Swapping to udp::socket: For chat or market‑data fan‑out, “best‑effort but immediate” sometimes wins. and TCP bakes in ordering, re‑transmission, and congestion avoidance; these guarantees come at the cost of extra context switches and kernel bookkeeping. udp::socket “best‑effort but immediate” #include #include #include using boost::asio::ip::udp; class UdpEchoServer { udp::socket socket_; std::array buf_{}; udp::endpoint remote_; public: explicit UdpEchoServer(boost::asio::io_context& io, unsigned short port) : socket_(io, udp::endpoint{udp::v4(), port}) { receive(); } private: void receive() { socket_.async_receive_from( boost::asio::buffer(buf_), remote_, [this](auto ec, std::size_t n) { if (!ec) send(n); }); } void send(std::size_t n) { socket_.async_send_to( boost::asio::buffer(buf_, n), remote_, [this](auto /*ec*/, std::size_t /*n*/) { receive(); }); } }; int main() { try { boost::asio::io_context io; UdpEchoServer srv(io, 9001); std::cout #include #include using boost::asio::ip::udp; class UdpEchoServer { udp::socket socket_; std::array buf_{}; udp::endpoint remote_; public: explicit UdpEchoServer(boost::asio::io_context& io, unsigned short port) : socket_(io, udp::endpoint{udp::v4(), port}) { receive(); } private: void receive() { socket_.async_receive_from( boost::asio::buffer(buf_), remote_, [this](auto ec, std::size_t n) { if (!ec) send(n); }); } void send(std::size_t n) { socket_.async_send_to( boost::asio::buffer(buf_, n), remote_, [this](auto /*ec*/, std::size_t /*n*/) { receive(); }); } }; int main() { try { boost::asio::io_context io; UdpEchoServer srv(io, 9001); std::cout << "⚡ UDP echo on 0.0.0.0:9001\n"; io.run(); } catch (const std::exception& ex) { std::cerr << ex.what() << '\n'; return 1; } } Latency table (localhost, 64‑byte payload): Latency table (localhost, 64‑byte payload): Layer TCP UDP Conn setup 3‑way handshake 0 HOL blocking Yes No Kernel buffer per‑socket shared RTT (median) ≈ 85 µs ≈ 45 µs Layer TCP UDP Conn setup 3‑way handshake 0 HOL blocking Yes No Kernel buffer per‑socket shared RTT (median) ≈ 85 µs ≈ 45 µs Layer TCP UDP Layer Layer TCP TCP UDP UDP Conn setup 3‑way handshake 0 Conn setup Conn setup 3‑way handshake 3‑way handshake 0 0 HOL blocking Yes No HOL blocking HOL blocking Yes Yes No No Kernel buffer per‑socket shared Kernel buffer Kernel buffer per‑socket per‑socket shared shared RTT (median) ≈ 85 µs ≈ 45 µs RTT (median) RTT (median) ≈ 85 µs ≈ 85 µs ≈ 85 µs ≈ 45 µs ≈ 45 µs ≈ 45 µs Here we replaced tcp::socket with udp::socket and removed the per‑session heap allocation; the code path is ~40 % shorter in perf traces. tcp::socket udp::socket If your application can tolerate an occasional drop (or do its own acks), UDP is the gateway to sub‑50 µs median latencies—even before kernel‑bypass. If you can tolerate packet loss (or roll your own ACK/NACK), UDP buys you ~40 µs on the spot. If your application can tolerate an occasional drop (or do its own acks), UDP is the gateway to sub‑50 µs median latencies—even before kernel‑bypass. If you can tolerate packet loss (or roll your own ACK/NACK), UDP buys you ~40 µs on the spot. I Takeaway: if you can tolerate packet loss (or roll your own ACK/NACK), UDP buys you ~40 µs on the spot. Takeaway: if you can tolerate packet loss (or roll your own ACK/NACK), UDP buys you ~40 µs on the spot. Takeaway: 3 – io_uring: The Lowest‑Friction Doorway to Zero‑Copy io_uring Linux 5.1 introduced io_uring; by 5.19 it rivals DPDK‑style bypass while staying in‑kernel. io_uring Avoids per‑syscall overhead by batching accept/recv/send in a single submission queue. Reuses a pre‑allocated ConnData buffer—no heap churn on the fast path. Achieves ~20 µs RTT on Apple M2‑>QEMU→Ubuntu, a 3× improvement over Boost.Asio/TCP (~85 µs). Avoids per‑syscall overhead by batching accept/recv/send in a single submission queue. Avoids per‑syscall overhead by batching accept/recv/send in a single submission queue. Reuses a pre‑allocated ConnData buffer—no heap churn on the fast path. Reuses a pre‑allocated ConnData buffer—no heap churn on the fast path. ConnData Achieves ~20 µs RTT on Apple M2‑>QEMU→Ubuntu, a 3× improvement over Boost.Asio/TCP (~85 µs). Achieves ~20 µs RTT on Apple M2‑>QEMU→Ubuntu, a 3× improvement over Boost.Asio/TCP (~85 µs). ~20 µs RTT // Extremely small io_uring TCP echo server (edge‑triggered) #include #include #include #include #include #include #include // --------------------------------------------------------------------------- // Compat shim for old liburing ( res) #endif constexpr uint16_t PORT = 9002; constexpr unsigned QUEUE_DEPTH = 256; constexpr unsigned BUF_SZ = 4096; struct ConnData { int fd; char buf[BUF_SZ]; }; int main() { // 1. Classic BSD socket setup int listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0); sockaddr_in addr{}; addr.sin_family = AF_INET; addr.sin_port = htons(PORT); addr.sin_addr.s_addr = INADDR_ANY; bind(listen_fd, reinterpret_cast (&addr), sizeof(addr)); listen(listen_fd, SOMAXCONN); // 2. uring setup io_uring ring{}; io_uring_queue_init(QUEUE_DEPTH, &ring, 0); // helper lambda: submit an accept sqe auto prep_accept = [&]() { io_uring_sqe* sqe = io_uring_get_sqe(&ring); sockaddr_in* client = new sockaddr_in; socklen_t* len = new socklen_t(sizeof(sockaddr_in)); io_uring_prep_accept(sqe, listen_fd, reinterpret_cast (client), len, SOCK_NONBLOCK); io_uring_sqe_set_data(sqe, client); // stash ptr so we can free later }; prep_accept(); io_uring_submit(&ring); std::cout << "⚡ io_uring TCP echo on 0.0.0.0:" = 0 && op (data); // free sockaddr io_uring_cqe_seen(&ring, cqe); // schedule next accept right away prep_accept(); // schedule first read ConnData* cd = new ConnData{client_fd, {}}; io_uring_sqe* r_sqe = io_uring_get_sqe(&ring); io_uring_prep_recv(r_sqe, client_fd, cd->buf, BUF_SZ, 0); io_uring_sqe_set_data(r_sqe, cd); io_uring_submit(&ring); continue; } // Read completed → if >0 bytes, write them back ConnData* cd = static_cast (data); if (op > 0) { io_uring_sqe* w_sqe = io_uring_get_sqe(&ring); io_uring_prep_send(w_sqe, cd->fd, cd->buf, op, 0); io_uring_sqe_set_data(w_sqe, cd); // reuse struct io_uring_submit(&ring); } else { // client closed close(cd->fd); delete cd; } io_uring_cqe_seen(&ring, cqe); } close(listen_fd); io_uring_queue_exit(&ring); return 0; } // Extremely small io_uring TCP echo server (edge‑triggered) #include #include #include #include #include #include #include // --------------------------------------------------------------------------- // Compat shim for old liburing ( res) #endif constexpr uint16_t PORT = 9002; constexpr unsigned QUEUE_DEPTH = 256; constexpr unsigned BUF_SZ = 4096; struct ConnData { int fd; char buf[BUF_SZ]; }; int main() { // 1. Classic BSD socket setup int listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0); sockaddr_in addr{}; addr.sin_family = AF_INET; addr.sin_port = htons(PORT); addr.sin_addr.s_addr = INADDR_ANY; bind(listen_fd, reinterpret_cast (&addr), sizeof(addr)); listen(listen_fd, SOMAXCONN); // 2. uring setup io_uring ring{}; io_uring_queue_init(QUEUE_DEPTH, &ring, 0); // helper lambda: submit an accept sqe auto prep_accept = [&]() { io_uring_sqe* sqe = io_uring_get_sqe(&ring); sockaddr_in* client = new sockaddr_in; socklen_t* len = new socklen_t(sizeof(sockaddr_in)); io_uring_prep_accept(sqe, listen_fd, reinterpret_cast (client), len, SOCK_NONBLOCK); io_uring_sqe_set_data(sqe, client); // stash ptr so we can free later }; prep_accept(); io_uring_submit(&ring); std::cout << "⚡ io_uring TCP echo on 0.0.0.0:" = 0 && op (data); // free sockaddr io_uring_cqe_seen(&ring, cqe); // schedule next accept right away prep_accept(); // schedule first read ConnData* cd = new ConnData{client_fd, {}}; io_uring_sqe* r_sqe = io_uring_get_sqe(&ring); io_uring_prep_recv(r_sqe, client_fd, cd->buf, BUF_SZ, 0); io_uring_sqe_set_data(r_sqe, cd); io_uring_submit(&ring); continue; } // Read completed → if >0 bytes, write them back ConnData* cd = static_cast (data); if (op > 0) { io_uring_sqe* w_sqe = io_uring_get_sqe(&ring); io_uring_prep_send(w_sqe, cd->fd, cd->buf, op, 0); io_uring_sqe_set_data(w_sqe, cd); // reuse struct io_uring_submit(&ring); } else { // client closed close(cd->fd); delete cd; } io_uring_cqe_seen(&ring, cqe); } close(listen_fd); io_uring_queue_exit(&ring); return 0; } Even without privileged NIC drivers, io_uring brings sub‑50 µs latency into laptop‑class hardware—ideal for prototyping HFT engines before deploying on SO_REUSEPORT + XDP in production. io_uring 4 – Running Benchmarks: Quantifying the wins We wrap each variant into Google Benchmarks #include #include #include #include using boost::asio::ip::tcp; using boost::asio::ip::udp; /* ---------- Helpers ------------------------------------------------------ */ // blocking Boost.Asio TCP echo client (loop‑back) static void tcp_roundtrip(size_t payload) { boost::asio::io_context io; tcp::socket c(io); c.connect({boost::asio::ip::make_address("127.0.0.1"), 9000}); std::string msg(payload, 'x'); c.write_some(boost::asio::buffer(msg)); std::array buf{}; c.read_some(boost::asio::buffer(buf, payload)); } // blocking Boost.Asio UDP echo client static void udp_roundtrip(size_t payload) { boost::asio::io_context io; udp::socket s(io, udp::v4()); udp::endpoint server(boost::asio::ip::make_address("127.0.0.1"), 9001); std::string msg(payload, 'x'); s.send_to(boost::asio::buffer(msg), server); std::array buf{}; s.receive_from(boost::asio::buffer(buf, payload), server); } #if defined(__linux__) // tiny wrapper for the io_uring server (assumes it’s already running on 9002) static void uring_tcp_roundtrip(size_t payload) { boost::asio::io_context io; tcp::socket c(io); c.connect({boost::asio::ip::make_address("127.0.0.1"), 9002}); std::string msg(payload, 'x'); c.write_some(boost::asio::buffer(msg)); std::array buf{}; c.read_some(boost::asio::buffer(buf, payload)); } #endif /* ---------- Benchmarks --------------------------------------------------- */ static void BM_AsioTCP_64B(benchmark::State& s) { for (auto _ : s) tcp_roundtrip(64); } BENCHMARK(BM_AsioTCP_64B)->Unit(benchmark::kMicrosecond); static void BM_AsioUDP_64B(benchmark::State& s) { for (auto _ : s) udp_roundtrip(64); } BENCHMARK(BM_AsioUDP_64B)->Unit(benchmark::kMicrosecond); #if defined(__linux__) static void BM_IouringTCP_64B(benchmark::State& s) { for (auto _ : s) uring_tcp_roundtrip(64); } BENCHMARK(BM_IouringTCP_64B)->Unit(benchmark::kMicrosecond); #endif BENCHMARK_MAIN(); #include #include #include #include using boost::asio::ip::tcp; using boost::asio::ip::udp; /* ---------- Helpers ------------------------------------------------------ */ // blocking Boost.Asio TCP echo client (loop‑back) static void tcp_roundtrip(size_t payload) { boost::asio::io_context io; tcp::socket c(io); c.connect({boost::asio::ip::make_address("127.0.0.1"), 9000}); std::string msg(payload, 'x'); c.write_some(boost::asio::buffer(msg)); std::array buf{}; c.read_some(boost::asio::buffer(buf, payload)); } // blocking Boost.Asio UDP echo client static void udp_roundtrip(size_t payload) { boost::asio::io_context io; udp::socket s(io, udp::v4()); udp::endpoint server(boost::asio::ip::make_address("127.0.0.1"), 9001); std::string msg(payload, 'x'); s.send_to(boost::asio::buffer(msg), server); std::array buf{}; s.receive_from(boost::asio::buffer(buf, payload), server); } #if defined(__linux__) // tiny wrapper for the io_uring server (assumes it’s already running on 9002) static void uring_tcp_roundtrip(size_t payload) { boost::asio::io_context io; tcp::socket c(io); c.connect({boost::asio::ip::make_address("127.0.0.1"), 9002}); std::string msg(payload, 'x'); c.write_some(boost::asio::buffer(msg)); std::array buf{}; c.read_some(boost::asio::buffer(buf, payload)); } #endif /* ---------- Benchmarks --------------------------------------------------- */ static void BM_AsioTCP_64B(benchmark::State& s) { for (auto _ : s) tcp_roundtrip(64); } BENCHMARK(BM_AsioTCP_64B)->Unit(benchmark::kMicrosecond); static void BM_AsioUDP_64B(benchmark::State& s) { for (auto _ : s) udp_roundtrip(64); } BENCHMARK(BM_AsioUDP_64B)->Unit(benchmark::kMicrosecond); #if defined(__linux__) static void BM_IouringTCP_64B(benchmark::State& s) { for (auto _ : s) uring_tcp_roundtrip(64); } BENCHMARK(BM_IouringTCP_64B)->Unit(benchmark::kMicrosecond); #endif BENCHMARK_MAIN(); With Google Benchmark we measured 10 K in‑process round trips per transport on an M2‑Pro MBP (macOS 14.5, Docker Desktop 4.30): Table 1 – Median RTT (64 B payload, 10 K iterations) Table 1 – Median RTT (64 B payload, 10 K iterations) Transport Median RTT (µs) Boost.Asio / TCP 82 Boost.Asio / UDP 38 io_uring / TCP 21 Transport Median RTT (µs) Boost.Asio / TCP 82 Boost.Asio / UDP 38 io_uring / TCP 21 Transport Median RTT (µs) Transport Transport Median RTT (µs) Median RTT (µs) Boost.Asio / TCP 82 Boost.Asio / TCP Boost.Asio / TCP 82 82 82 Boost.Asio / UDP 38 Boost.Asio / UDP Boost.Asio / UDP 38 38 38 io_uring / TCP 21 io_uring / TCP io_uring / TCP io_uring 21 21 21 Even on consumer hardware, io_uring halves UDP’s latency and crushes traditional TCP by nearly 4×. This validates the architectural decision to build NimbusNet’s high‑fan‑out chat tier on kernel‑bypass primitives while retaining a pure‑userspace codebase. io_uring 4× Takeaways & Future Work Portability first, performance second pays dividends—macOS dev loop, prod Linux wins. UDP is “good enough” for most chats; sprinkle FEC / acks for mission‑critical flows. io_uring slashes latency without root privileges, making kernel‑bypass approachable. Portability first, performance second pays dividends—macOS dev loop, prod Linux wins. Portability first, performance second UDP is “good enough” for most chats; sprinkle FEC / acks for mission‑critical flows. io_uring slashes latency without root privileges, making kernel‑bypass approachable. io_uring Next steps SO_REUSEPORT + sharded accept rings → horizontal scale on 64‑core EPYC Processor TLS off‑loading via kTLS with io_uring::splice. eBPF tracing to pinpoint queue depth vs. tail latency. SO_REUSEPORT + sharded accept rings → horizontal scale on 64‑core EPYC Processor SO_REUSEPORT + sharded accept rings TLS off‑loading via kTLS with io_uring::splice. TLS off‑loading kTLS io_uring::splice eBPF tracing to pinpoint queue depth vs. tail latency. eBPF tracing