ann.md

Conduit 3.0.0

I'm glad to announce the new release of [conduit][conduit]. Conduit is, technically, a framework which wants to provide an abstraction of protocols. It permits to implement free-dependencies protocols.

Introduction

The goal of Conduit is the abstraction of protocols. In OCaml, it exists several ways to abstract an implementation from something else. By this way, it should be easy, for example, to abstract a TLS implementation from an HTTP implementation.

However, such choice can be deep in the stack - and it moves to one step more the way to get your HTTP implementation systematically (by functors for example). This problematic (the functors hell) is well-known for MirageOS which wants to abstract... everything! So it's why Conduit exists.

The idea is to use Conduit as the protocol implementation. But it's not! It lets the end-user to choose then which protocol should be used/injected behind Conduit.

By this abstraction mechanism, a protocol implementer can get rid the responsibility of the protocols choice (like which TLS implementation we should use between OpenSSL or our great ocaml-tls library). Then, the user can choose/compose protocols and inject them into Conduit (depending on its own context - like a MirageOS context).

Usage of Conduit

Such framework allows us to separate the logic of a protocol from underlying implementation needed to communicate with a peer. The distribution of Conduit comes with [a simple tutorial][howto] which explains step by step how to implement a ping-pong client & server and, the most important, how to upgrade them with TLS.

With Conduit, we ensure the compatibility with MirageOS (and specially mirage-tcpip) while being usable by others. Of course, Conduit is not mandatory to ensure this compatibility but it helps us for higher libraries such as [ocaml-git][ocaml-git]/[Irmin][irmin] or [Cohttp][cohttp].

Concrete improvements

Abstract and destruct it!

The most requested feature on the new version of Conduit is the ability to destruct the [Conduit.flow][conduit-flow]. The ability to abstract the protocol comes with the abstract type Conduit.flow. The new version permits to destruct it to a well-known value (such as an UNIX socket):

let handler flow = match flow with
  | Conduit_lwt.TCP.T (Value file_descr) ->
    let peer = Lwt_unix.getpeername file_descr in
    ...
  | flow -> ... (* other kind of protocol *)
  
let run =
  Cohttp_lwt_unix.serve ~handler
    { sockaddr= Unix.inet_addr_loopback }

The dispatch of the protocol

The other most interesting feature of Conduit is the full control of the dispatch between protocols by the end-user. From a concrete information such as an Uri.t, the end-user is able to describe how Conduit should choose the protocol (and with which value it should try to initiate the connection):

let my_tls_config = Tls.Config.client ...

let connect uri =
  let edn = Conduit.Endpoint.of_string
    (Uri.host_with_default ~default:"localhost" uri) in
  let resolvers = match Uri.scheme uri with
    | Some "https" ->
      let port = Option.value ~default:443 (Uri.port uri) in
      Conduit_lwt.add
        Conduit_lwt_tls.TCP.protocol
        (Conduit_lwt_tls.TCP.resolve ~port ~config:my_tls_config)
        Conduit.empty
    | Some "http" | None ->
      let port = Option.value ~default:80 (Uri.port uri) in
      Conduit_lwt.add
        Conduit_lwt.TCP.protocol
        (Conduit_lwt.TCP.resolve ~port)
        Conduit.empty in
  Conduit_lwt.resolve ~resolvers edn >>= fun flow ->
  ...

An explicit way to launch a server

Conduit comes with a new API about the server-side where anything become explicit: no dispatch, no hidden choice. It proposes now a simple function to start the usual server loop:

let run handler =
  Conduit_lwt.serve ~handler
    Conduit_lwt.TCP.service
    { Conduit_lwt.TCP.sockaddr= Unix.(ADDR_INET (inet_addr_loopback, 8080)
    ; capacity= 40 }

Conclusion

Conduit is a piece required by many libraries but nobody really use it. This new version wants to replace and redefine more concretely what Conduit is. The update is [huge][conduit-pr] for us but small for people where we tried to keep the same global idea of the abstraction.

I would like to thank many people (MirageOS core team, Cohttp peoples, some not so famous guys of the Reason/OCaml eco-system) who follow us on this deep development (and tried and iterated on our version). It does not change too much our world, but it paves the way for a better MirageOS/OCaml eco-system.

As a french guy, I just would like to say: Conduit est mort, Vive Conduit!

[howto]: [conduit]: https://github.com/mirage/ocaml-conduit [cohttp]: https://github.com/mirage/ocaml-cohttp [ocaml-git]: https://github.com/mirage/ocaml-git [irmin]: https://github.com/mirage/irmin [conduit-pr]: mirage/ocaml-conduit#311