CNI plugin chaining discussion

plugin-chaining.md

CNI plugin chaining proposals

Motivation

CNI does not currently have a defined way to chain plugins. There is implicit chaning via the IPAM mechanism, where it is assumed that the top-level plugin (e.g. macvlan) will find and execute an IPAM plugin, but this is not made more generic.

Motivating examples

Flannel

The flannel plugin is runtime-interface-agnostic. It accepts an interface configuration, modifies it based on runtime flannel state, and passes this to the desired "real" plugin.

tuning

The sysctl plugin does not currently chain. Instead, it modifies syctl tunables in the runtime's namespace

port-forwarding

A port-forwarding plugin creates iptables rules on the host. It needs to know the runtime's IP, but no interface details

Traffic shaping

Traffic shaping needs to add tc rules on the "host" interface. When multiple host interfaces are created, the behavior needs to be consistent.

Proposals:

Inside-out

Correct ordering is up to the user. Plugins that receive an inner block but do not expect it MUST error.

{
  "type": "tuning",
  "sysctl": {
    "net.netfilter.nf_conntrack_max": "524288"
  },
  "inner": {
    "type": "flannel",
    "inner": {
      "type": "bridge",
      "mtu": "4000",
      "ipam": {
      }
    }
  }
}

Ordered

{
  "name": "awesome-network",
  "plugins": [
    {
      "type": "host-local",
      "subnet": "10.0.0.1/24"
    },
    {
      "type": "bridge",
      "mtu": 1400,
    },
    {
      "type": "port-forwarding",
      "ports": {"20.0.0.1:8080": "80"}
    }
  ]
}

Multi-call

Rather than specifying the complete network configuration as a single file (and therefore constructing the network in a single operation), provide a calling convention for post-interface plugins. Specifically, the runtime (or libcni) should provide the results of the previous plugin as part of the configuration.

{
  "name": "mynet",
  "type": "macvlan",
  "ipam": {
    "type": "dhcp"
  }
}

Then, the runtime may issue further calls using a separate configuration file, including the response from the previous plugin execution.

{
  "name": "tuning",
  "sysctl": {
    "net.netfilter.nf_conntrack_max": "524288"
  },
  "result": {
    "interfaces": [ ],
    "ip": [ ]
  }
}

This allows for network definition to be kept relatively static, whereas the simpler tuning or firewall configuration can be more easily generated on-the-fly. The runtime would be responsible for calling DELETE appropriately.

Structured

The Setup plugin is essentialy ipam, but it has the ability to edit the parsed configuration. Multiple setup plugins are permitted. In this case, "flannel" would be a setup plugin.

(firewall configuration is just a placeholder)

{
  "name": "my-flannel-network",
  "setup": [
    {"name": "flannel"}
  ],
  "if": {
    "type": "ptp",
    "ipMasq": true
  },
  "plugins": [
    {
      "type": "shaping",
      "up": "1mbps",
      "down": "2mbps"
    },
    {
      "type": "firewall",
      "allow": "10.0.0.0/8",
      "deny": "any"
    }
  ]
}