egernst/k8s-play.md

## k8s-play.md

      
    Raw
  

              k8s-play.md
            
          
    Get go 1.8.3:

wget https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz
sudo tar -xvf go1.8.3.linux-amd64.tar.gz -C /usr/local/
mkdir -p $HOME/go/src
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
go version

Build CRI-O from source:

sudo apt-get install -y \
  autoconf \
  btrfs-tools \
  git \
  libapparmor-dev \
  libassuan-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libc6-dev \
  libgpgme11-dev \
  libgpg-error-dev \
  libseccomp-dev \
  libselinux1-dev \
  pkg-config

sudo add-apt-repository ppa:alexlarsson/flatpak
sudo apt-get update

sudo apt-get install -y libostree-dev


go get -d github.com/kubernetes-incubator/cri-o
cd $GOPATH/src/github.com/kubernetes-incubator/cri-o
make install.tools
make
sudo make install
sudo make install.config

Install latest RUNC:

go get github.com/opencontainers/runc
cd $GOPATH/src/github.com/opencontainers/runc
make
sudo make install
runc --version

This will install runc at usr/local/sbin/runc.  CRIO expects it at /usr/bin/runc.
Either move the binary to the expected path or modify /etc/crio/crio.conf to point to
the installed location.
Get CNI Plugins from source:


go get -d github.com/containernetworking/plugins
cd $GOPATH/src/github.com/containernetworking/plugins
git checkout dcf7368eeab15e2affc6256f0bb1e84dd46a34de
./build.sh
sudo mkdir -p /opt/cni/bin
sudo cp bin/* /opt/cni/bin/

Configure CNI:

sudo mkdir -p /etc/cni/net.d

sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
    "cniVersion": "0.2.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0"  }
        ]
    }
}
EOF'

sudo sh -c 'cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
    "cniVersion": "0.2.0",
    "type": "loopback"
}
EOF'

Install Clear Containers Runtime

For Clear Containers 3.0, follow directions available at https://github.com/clearcontainers/runtime/blob/master/docs/developers-clear-containers-install.md
Modify CRI-O configuration to make use of Clear Containers:

Modify /etc/crio/crio.conf to select cc-runtime as the runtime_untrusted_workload
and set the default_workload_trust to untrusted.
runtime_untrusted_workload = "/usr/local/bin/cc-runtime"
default_workload_trust = "untrusted"

Start CRI-O System Daemon

Note the Environment parameters set below for proxy which will need to be updated if you are operating behind a proxy.
# sh -c 'echo "[Unit]
Description=OCI-based implementation of Kubernetes Container Runtime Interface
Documentation=https://github.com/kubernetes-incubator/cri-o

[Service]
ExecStart=/usr/local/bin/crio --debug
Environment="HTTP_PROXY=http://myproxy.example.com:8080" "NO_PROXY=example.com,.example.com,localhost"
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target" > /etc/systemd/system/crio.service'

sudo systemctl daemon-reload
sudo systemctl enable crio
sudo systemctl start crio
sudo crioctl runtimeversion

Verify CRIO is working appropriately

Make sure you can setup a POD and container:
cd $GOPATH/src/github.com/kubernetes-incubator/cri-o
sudo mkdir -p /etc/containers
sudo cp test/policy.json /etc/containers/policy.json 
POD_ID=$(sudo crioctl pod run --config test/testdata/sandbox_config.json)
sudo crioctl pod status --id $POD_ID
sudo crioctl image pull redis:alpine
CONTAINER_ID=$(sudo crioctl ctr create --pod $POD_ID --config test/testdata/container_redis.json)
sudo crioctl ctr start --id $CONTAINER_ID
sudo crioctl ctr status --id $CONTAINER_ID

Clean up:
sudo crioctl ctr stop --id $CONTAINER_ID
sudo crioctl ctr remove --id $CONTAINER_ID
sudo crioctl pod stop --id $POD_ID
sudo crioctl pod remove --id $POD_ID
sudo crioctl pod list
sudo crioctl ctr list

Or, run the bat test:
sudo apt-get install -y bats
sudo -E PATH=$PATH RUNTIME=cc-runtime ./test/test_runner.sh TESTFLAGS="test/ctr.bats"

Setup Kubernetes to Use our CRI-O Setup

After verifying CRI-O is working expected in a standalone setup, install and setup K8S to make use of it.
Install Kubernetes


Install the 1.6.7 binaries:

# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main
EOF
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# apt-get update
# apt-get install -y docker.io kubelet=1.6.7-00 kubeadm=1.6.7-00 kubectl=1.6.7-00 
# sudo apt-mark hold kubelet kubeadm kubectl


Modify the default systemd file to make use of CRI:

Modify /etc/systemd/system/kubelet.service.d/10-kubeadm.conf to add the following parameters to kubelet:
--container-runtime=remote --container-runtime-endpoint=/var/run/crio.sock --runtime-request-timeout=15m

An example for reference of the full file is:
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/crio.sock --runtime-request-timeout=30m"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS

# systemctl daemon-reload


Start the master:

kubeadm init --pod-network-cidr 10.244.0.0/16 && export KUBECONFIG=/etc/kubernetes/admin.conf


Setup flannel:

curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sudo -E kubectl create -f kube-flannel-rbac.yml
sudo -E kubectl get nodes
sudo -E kubectl get pods
sudo -E kubectl get pods --all-namespaces
sudo -E kubectl create --namespace kube-system -f kube-flannel.yml


Cleanup

sudo -E kubeadm reset
for c in `sudo crioctl ctr list | grep ^ID | cut -c5-`; do sudo crioctl ctr stop --id $c; sudo crioctl ctr remove --id $c ; done
for c in `sudo crioctl pod list | grep ^ID | cut -c5-`; do sudo crioctl pod stop --id $c; sudo crioctl pod remove --id $c ; done