Skip to content

Instantly share code, notes, and snippets.

@cbodley
Created March 6, 2023 16:29
Show Gist options
  • Save cbodley/77bf4adf03998908fbc75e7551af14bf to your computer and use it in GitHub Desktop.
Save cbodley/77bf4adf03998908fbc75e7551af14bf to your computer and use it in GitHub Desktop.
asio and coroutines in ceph

contents

  1. what is asio?
  2. asio::io_context as a work queue
  3. CompletionToken concept
  4. Executor concept
  5. c++20 coroutines
  6. asio in ceph
  • a networking library

    • abstractions for sockets, streams, buffers
  • flexible asynchronous runtime

    • can be single- or multithreaded
    • async operations can complete via callbacks, futures, coroutines, etc
  • header-only c++ template library

excellent documentation: see overview and reference

example: post() multiple handlers

asio::io_context ctx;

// schedule some work
asio::post(ctx, [] { std::cout << "hello "; });
asio::post(ctx, [] { std::cout << "world!"; });

ctx.run(); // run the work -> "hello world!"

example: connect to a remote tcp server

asio::io_context ctx;
asio::ip::tcp::socket socket{ctx};
    
// initiate a socket connect operation
socket.async_connect(endpoint,
    [&] (error_code ec) {
      if (ec) {
        std::cerr << "connect failed with " << ec.message() << std::endl;
      } else {
        std::cout << "connected to " << endpoint << std::endl;
      }
    });

ctx.run(); // run until the operation completes

takeaways

  • io_context::run() blocks until all work is done
  • a single thread can run multiple concurrent asynchronous jobs

each async function in asio takes a templated CompletionToken as its final argument. this specifies how the caller should be signaled on the operation's completion

the simplest kind of CompletionToken is a callback function, like the lambda functions we saw in the examples

each async operation in asio documents its completion signature: see tcp::socket::async_connect()

see doc for future and coroutine examples

CompletionToken adapters

asio provides several functions that modify the behavior of a wrapped CompletionToken:

an executor is a handle to an execution context. asio::io_context::get_executor() returns a handle of type asio::io_context::executor_type

each of asio's io objects, like the sockets and timers, take the Executor both as a template parameter and as a constructor argument. the io object exposes this default executor with a get_executor() member function

when you call an async member function on the io object, the CompletionToken will be associated with this default executor. this associated executor can be overridden with asio::bind_executor() to run the completion somewhere else

asio::strand<Executor> (doc)

a special kind of executor that wraps another executor with an additional guarantee that only one of its handlers will run at a time. a single-threaded execution context already guarantees this, so they're mainly useful in multi-threaded contexts

asio::thread_pool ctx{4}; // execution context with 4 threads

auto strand = asio::make_strand(ctx);

asio::post(strand, [] { std::cout << "hello "; });
asio::post(strand, [] { std::cout << "world!"; });

ctx.join(); // run the work -> "hello world!"

with careful use of strands, a multithreaded application can safely share memory between handlers without the need for any explicit locking

asio::any_io_executor (doc)

to write code that works for any kind of execution context or strand, you'd normally have to template everything on the Executor type. these templates can be avoided by using the polymorphic asio::any_io_executor which can hold any kind of Executor

a c++20 coroutine is any function that contains one of the co_await, co_yield, or co_return keywords

asio::awaitable<T> (doc)

the return type of c++20 coroutine functions that can be run on asio executors

asio::awaitable<int> answer_to_life_the_universe_and_everything()
{
  co_return 42;
}

asio::co_spawn() (doc)

spawns a new coroutine-based 'thread' of execution

asio::io_context ctx;
auto ex = ctx.get_executor();

asio::co_spawn(ex, answer_to_life_the_universe_and_everything(),
    [] (std::exception_ptr eptr, int answer) {
      if (eptr) {
        std::rethrow_exception(eptr);
      } else {
        std::cout << "the answer is " << answer;
      }
    });

ctx.run(); // run the coroutine -> "the answer is 42"

once spawned, a coroutine can co_await calls to other coroutines:

asio::awaitable<void> child();

asio::awaitable<void> parent()
{
  co_await child();
}

asio::use_awaitable (doc)

the CompletionToken a coroutine function uses to wait on asynchronous operations

asio::awaitable<void> connect(tcp::socket& socket, tcp::endpoint remote)
{
  co_await socket.async_connect(remote, asio::use_awaitable);
}

asio::this_coro::executor (doc)

allows a coroutine function to query its own executor

asio::awaitable<void> sleep_for(ceph::timespan duration)
{
  // query the executor using co_await
  auto ex = co_await asio::this_coro::executor;

  // construct a timer on the same executor
  asio::steady_timer timer{ex, duration};
  
  // wait on the timer
  co_await timer.async_wait(asio::use_awaitable);
}

putting it all together

coroutine echo server example: https://www.boost.org/doc/libs/1_79_0/doc/html/boost_asio/example/cpp17/coroutines_ts/echo_server.cpp

asio in ceph

rgw's beast frontend (source)

  • creates a pool of rgw_thread_pool_size threads that each call io_context::run()
  • uses asio tcp::sockets for all network io
  • timers for connection timeouts
  • spawns a stackful coroutine (yield_context) to handle each http connection
  • uses yield_context as the CompletionToken for all of its async operations

neorados client library (source)

  • class neorados::RADOS is an io object that exposes the entire librados API as asynchronous operations taking CompletionToken

librbd

uses asio and neorados

conclusion and questions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment