egernst

Using a Docker CNM plugin to play with SRIO-V

This gist describes the setup necessary for testing SRIO-V based connectivity between two physical boxes which are each setup as described here, and directly connected via their respective SRIO-V enabled NICs.

Setup host system's packages

For this scenario, I'm setting up two Ubuntu 16.04 systems which have a SRIO-V enabled interface as well as a second port for accessing the SUT. To setup:

egernst

 sudo crictl runs ./sandbox_config.json
 sudo crictl create db1409f8ef944cd2a2cb07 container_bad.json sandbox_config.json 
 sudo crictl sandboxes
 sudo crictl ps

egernst

161c161
< CONFIG_CGROUP_RDMA=y
---
> # CONFIG_CGROUP_RDMA is not set
356d355
< CONFIG_BLK_MQ_RDMA=y
736d734
< # CONFIG_SMC is not set

egernst

root@eernstworkstation:/var/run/docker/netns# iptables-save

Generated by iptables-save v1.6.0 on Fri Mar 2 14:08:00 2018

*nat :PREROUTING ACCEPT [1:60] :INPUT ACCEPT [1:60] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :DOCKER_OUTPUT - [0:0] :DOCKER_POSTROUTING - [0:0] -A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT

egernst

Title: Kata Flow
participant CRI
participant CRIO
participant Kata Runtime
participant virtcontainers
participant hypervisor
participant agent
participant shim
participant proxy
#

egernst

sudo journalctl --rotate && sudo journalctl --vacuum-time=1seconds

egernst

apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.TrustedSandbox: "false"
  name: ubuntu-pod-untrusted
spec:
  containers:
  - name: ubuntu
    image: "ubuntu:14.04"

egernst

1. Allow port input on 443 fo the azure instance
2. Grab the cluster conf (.kube/config) to the local machine
3. Install kubectl on the local machine
4. Make sure you can access the cluster
5. On local, run kubectl proxy (this is specific to do port forwarding for the dashboard itself)
6. Connect to URL:
7. Get the token to use for logging in: kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

egernst

Getting Docker working well behind Proxy on Ubuntu

Docker's runtime config setup:

$ cat ~/.docker/config.json 
{
	"proxies":
	{
 "default":

egernst

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubelet-cri-o-kata-cleanup
  namespace: kube-system
spec:
  selector:
      matchLabels:
        name: kubelet-cri-o-kata-cleanup