updated launcher proposal

launcher.slide

External Chaincode Launcher
Roadmap

Matthew Sykes
IBM
sykesmat@us.ibm.com


* Why?

* Hard requirement on docker

Building and running chaincode requires a docker daemon. This is problematic in security conscious environments.

- A docker daemon makes it trivial to run arbitrary code on the host as a privileged user.
- A docker daemon requires elevated privileges to create and manage various kernel namespaces.

Within the context of Kubernetes

- Kubernetes no longer uses docker by default but peers still require docker.
- If docker is run within the pod hosting the peer, it violates recommended pod security policy.

* Complexity of adding new chaincode types

Support for different chaincode language types is implemented in the peer as _Platforms_.

A _Platform_ implementation must provide the following:

- Logic to package chaincode for the peer
- Logic to generate a `Dockerfile` capable of building the chaincode
- Docker images for building and running the chaincode
- Configuration changes for docker images and templates

Outside of the _Platform_, the peer needs additional modifications

- Declaring a new chaincode type in protobuf messages
- Updates to the cli for the chaincode type
- Updates to the chaincode runtime to generate a start command (not part of the platform)

* Background

* Chaincode Deployment

At a high level, the current chaincode deployment and execution model consists of the following steps:

- determine the chaincode type
- create a package that contains the source artifacts
- transfer the artifacts to the peer
- transform the source into something that can execute
- run the executable

We've seen this before...

* Buildpacks

A buildpack is software that transforms application source code into runnable artifacts.

Heroku, one of the first successful PaaS offerings, provided the buildpack specification to enable deployment of arbitrary 12-factor applications. A buildpack implements three executables:

- /bin/detect
- /bin/compile
- /bin/release

: Heroku defined the original buildpack contract
: CloudFoundry extended the buildpack contract with supply/finalize
: CNCF is hosting the buildpacks.io project to adapt to OCI images

* /bin/detect

`detect` is used to determine if a buildpack should be used.

Platforms iterate over a configured list of buildpacks. The first one that returns successfully from `detect` will be used to `compile` the application.

* /bin/compile

`compile` is used to transform the source into something that can run.

* /bin/release

`release` is used to provide metadata about how to run the compiled application.

In the original Heroku implementation, `release` would generate a YAML hash that included a `Procfile` that included the command and arguments required to run the executable.

* Goals

* Goals

- Remove the hard requirement on docker
- Extract and decouple chaincode packaging, building, and execution from the peer.


* Proposal

* Proposal

Adapt the buildpack model for chaincode and extend it to include a `run` operation that is responsible for executing the chaincode.

This model will introduce seams that enable operators to build and launch chaincode with the technology of their choice and, eventually, will enable us to remove hard coded chaincode platforms from the peer.

* External Chaincode Launcher

An external chaincode launcher provides the following executables:

- /bin/detect
- /bin/build
- /bin/release
- /bin/run

* Chaincode /bin/detect

`detect` determines if the external builder should be used with the chaincode. If detect exits with a return code of 0, the external builder will be used to build the chaincode.

The executable will be called by the peer with two arguments:

- a directory containing the extracted contents of the chaincode code package
- a directory containing the `metadata.json` file from the chaincode package

A typical implementation of `detect` will inspect the contents of the chaincode package and/or metadata to determine if it is able to handle the compilation and execution of the chaincode. A simple implementation will examine the chaincode type information from `metadata.json`.

* Chaincode /bin/build

`build` is called to transform the chaincode code package into something that can be executed by `run`.

The executable is called with three arguments:

- a directory containing the extracted contents of the chaincode code package
- a directory containing the `metadata.json` file from the chaincode package
- a directory to be populated by `build` with the artifacts required to run the chaincode

A typical implementation of `build` will use a portable tool chain packaged with the buildpack to compile the chaincode.

* Chaincode /bin/release (optional)

`release` is called to generate metadata necessary to control how the peer starts the application.

The executable is called with two arguments:

- the directory that was populated by `build` with the artifacts required to run the chaincode
- a directory to be populated by `release` with a metadata and assets

The details have not been worked out but there are two kinds of information we're expecting:

- metadata like CouchDB indices that have been extracted from chaincode to be consumed by the peer
- information about the location and assets necessary for a peer to connect to running chaincode that will not be started by the peer

* Chaincode /bin/run (optional)

`run` is used to run chaincode if `release` did not provide connection information to running chaincode.

The executable is called with two arguments:

- the directory populated by `build` with the artifacts required to run the chaincode
- a directory containing a `chaincode.json` and any artifacts referenced by that file

`chaincode.json` will be populated by the peer with the information like the chaincode server address and TLS keys and certificates.

A typical implementation of `run` will start the application with the necessary environment and command line flags.


* Configuration

The peer configuration will be extended to include a list of external chaincode builders.

   chaincode:
     external_builders:
     - name: one
       path: /path/to/builder/one
       environment_whitelist:
       - PATH
       - LIBPATH
     - name: two
       path: /path/to/builder/two

- The `name` field provides a name that will be used in logs.
- The `path` field points to the root of the buildpack. This minimally contains a bin directory.

The detect phase will run through the list in the order specified. If none of the specified builders have detected the application, provided the docker endpoint is still configured, we will fallback to the existing docker based build process that is embedded within the peer.

This model preserves existing behavior while enabling experimentation.

* Interested Parties

- SGX enabled Fabric private chaincode
- IBP
- Fabric integration tests
- State Street "cloud native" chaincode