CNX INSTALL

Step 1: UCP install with --unmanaged-cni flag 

If you are relying on testkit 
```
#!/bin/bash
echo "Setting environment variables"
export NUM_LINUX_HOSTS=3
export NUM_WINDOWS_HOSTS=0
export HUB_NAMESPACE="dockereng"
export HUB_TAG="3.1.0-latest" (Please check with ucp team for the latest tag)
export TESTKIT_SKIP_SELINUX=1
export TESTKIT_STORAGE_DRIVER=overlay2
export TESTKIT_PLATFORM_LINUX=ubuntu_16.04
export TESTKIT_ENGINE="ee-test"
export UCP_ADMIN_PASSWORD="docker2018"
echo "Running testkit"
echo "testkit-darwin-amd64 create $NUM_LINUX_HOSTS $NUM_WINDOWS_HOSTS --ucp --parallel --org $HUB_NAMESPACE --tag $HUB_TAG --engine $TESTKIT_ENGINE --debug --args "--admin-password $UCP_ADMIN_PASSWORD --unmanaged-cni""
./testkit-darwin-amd64 create $NUM_LINUX_HOSTS $NUM_WINDOWS_HOSTS --ucp --parallel --org $HUB_NAMESPACE --tag $HUB_TAG --engine $TESTKIT_ENGINE --debug --args "--admin-password $UCP_ADMIN_PASSWORD --unmanaged-cni"
```

At the end of this step ucp will be installed on all nodes. No CNI plugins will be deployed. Now use the admin bundle in the
testkit output which has the admin client bundle. Alternatively you can get it by logging into the ucp url and downloading 
the client bundle. 

Get the etcd information from /var/lib/docker/volumes/ucp-node-certs/_data/. I have posted the sample values. Get the base64 encode
values of the "files" and populate as shown below for etcd-ca,etcd-cert etc.

Follow instructions for install instructions and rbac.yaml deployment from https://docs.tigera.io/v2.1/getting-started/kubernetes/installation/calico 
Use the below yaml for calico.yaml and not download from the location.  


```
# CNX Version v2.1.1
# https://docs.tigera.io/v2.1/releases#v2.1.1
# This manifest includes the following component versions:
#   calico/node:v2.1.1
#   calico/cni:v3.1.2
#   calico/kube-controllers:v3.1.2

# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://proxy.local:12378"

  # Configure the Calico backend to use.
  calico_backend: "bird"

  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "nodename": "abhi-testkit-3b67a1-ubuntu-0",
      "nodename_file_optional": true,
      "cniVersion": "0.3.0",
      "plugins": [
        {
          "type": "calico",
          "etcd_endpoints": "https://proxy.local:12378",
          "etcd_key_file": "/var/lib/docker/ucp/ucp-node-certs/key.pem",
          "etcd_cert_file": "/var/lib/docker/ucp/ucp-node-certs/cert.pem",
          "etcd_ca_cert_file": "/var/lib/docker/ucp/ucp-node-certs/ca.pem",
          "log_level": "info",
          "mtu": 1500,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "/var/lib/docker/ucp/ucp-node-certs/kubeconfig"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }

  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: "/calico-secrets/etcd-ca"
  etcd_cert: "/calico-secrets/etcd-cert"
  etcd_key: "/calico-secrets/etcd-key"

---

# The following contains k8s Secrets for use with a TLS enabled etcd cluster.
# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following files with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # This self-hosted install expects three files with the following names.  The values
  # should be base64 encoded strings of the entire contents of each file.
   etcd-ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJhekNDQVJDZ0F3SUJBZ0lVVkkvaFBwdXpESXk5dEUxT3pYNmVGdjUyWGRBd0NnWUlLb1pJemowRUF3SXcKRXpFUk1BOEdBMVVFQXhNSWMzZGhjbTB0WTJFd0hoY05NVGd3TmpFNU1UYzFPREF3V2hjTk16Z3dOakUwTVRjMQpPREF3V2pBVE1SRXdEd1lEVlFRREV3aHpkMkZ5YlMxallUQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VICkEwSUFCRER5aWZOb1h3WGVmcDNaWmN4bkNvNmQ3S3l1UCtSWkxiQjVWOXpVMFhWUEZJdmtQWmVMNlR2RElPL2UKK2ZlWkNIUm93V0JUdm83cXhJQXEzb3B2cGg2alFqQkFNQTRHQTFVZER3RUIvd1FFQXdJQkJqQVBCZ05WSFJNQgpBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlJiREtTWlpEaTN0TUxlTXBrZThGU2h0SjROVXpBS0JnZ3Foa2pPClBRUURBZ05KQURCR0FpRUF3TkhzVTlDcWlnQUVsYWcyU2lSR0xEQS9meUVySEcvcUF3REl6NmJPekRZQ0lRQy8KSk56OWVXTXpZY2FCeDE5b1IyOHpBMVJiQ0V6YXplODZPVTgyUHliYTR3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= 
   etcd-cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGVENDQXJ1Z0F3SUJBZ0lVZVA3cXF6TzhvSjI5aG0yOUxnMkE0YU5Cb3dNd0NnWUlLb1pJemowRUF3SXcKRXpFUk1BOEdBMVVFQXhNSWMzZGhjbTB0WTJFd0hoY05NVGd3TnpNeE1EQXhNVEF3V2hjTk1UZ3hNREk1TURBeApNVEF3V2pBL01RNHdEQVlEVlFRTEV3VnpkMkZ5YlRFdE1Dc0dBMVVFQXhNa2MzbHpkR1Z0T25WamNEcGpaM055CmRYRjBkbXMzYVcwd05tOTRNbU5zTW5NMU5uY3pNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUUKa0hZekM2LzFHUHBVTng0L3A5U3BZRHZWQzBTT1FqTVRsZ01ySUk4eVVSU09WZDd0dUZVOHd2bDV0N291UWYwQQpIWTZBTXJOM25VMEplcXl2WUdud1NxT0NBYjh3Z2dHN01BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFCkZqQVVCZ2dyQmdFRkJRY0RBUVlJS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWRCZ05WSFE0RUZnUVUKMFhwVjExeWZ0b3licVhnMGExR2x1RFFMRzdzd0h3WURWUjBqQkJnd0ZvQVVXd3lrbVdRNHQ3VEMzaktaSHZCVQpvYlNlRFZNd2dnRTZCZ05WSFJFRWdnRXhNSUlCTFlJTGNISnZlSGt1Ykc5allXeUNIR0ZpYUdrdGRHVnpkR3RwCmRDMHpZalkzWVRFdGRXSjFiblIxTFRDQ0pHdDFZbVZ5Ym1WMFpYTXVaR1ZtWVhWc2RDNXpkbU11WTJ4MWMzUmwKY2k1c2IyTmhiSUlLYTNWaVpYSnVaWFJsYzRJSmJHOWpZV3hvYjNOMGdodGpiMjF3YjNObExXRndhUzVyZFdKbApMWE41YzNSbGJTNXpkbU9DR1hrNGRIUm1aRFYwZDJFNWJqVXhaSGt4WTJWbGNuSm1NV0dDRWUrL3ZUVTBMakU0Ck9DNHhOVGN1TVRNd2doSnJkV0psY201bGRHVnpMbVJsWm1GMWJIU0NGbXQxWW1WeWJtVjBaWE11WkdWbVlYVnMKZEM1emRtT0hCQXBnQUFHSEJEYThuWUtIQkt3ZkVVR0hCQ09peHArSEJDTGNMc1dIQkNMV2NpZUhCS3dSQUFHSApCSDhBQUFHSEJEYktIbGVIQkNMVkFBR0hCRFpIbHJtSEJEYTdRVnFIQkRiVmVsNHdDZ1lJS29aSXpqMEVBd0lEClNBQXdSUUloQU5lcTMvR08wZHpRM09wOGNWRFdWcVdoTS9xNHY4aUxDTmNOdjRSU3ZtcUlBaUEzQy9hSVNGaHUKVzB0WXRjTTAyL3o4a29lQ2lPbkt0dGQrMHNlM0lsQUhPUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K 
   etcd-key: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUo0dmVqYWdFYlVldjlwalRPcHBTY2gwcVQvdkh1K243ZThURFBycXQzNDFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFa0hZekM2LzFHUHBVTng0L3A5U3BZRHZWQzBTT1FqTVRsZ01ySUk4eVVSU09WZDd0dUZVOAp3dmw1dDdvdVFmMEFIWTZBTXJOM25VMEplcXl2WUdud1NnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=
---

# This manifest installs the calico/node container, as well
# as the Calico CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      hostNetwork: true
      tolerations:
        # Make sure calico/node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      containers:
        # Runs calico/node container on each Kubernetes node.  This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: quay.io/tigera/cnx-node:v2.1.1
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set noderef for node controller.
            - name: CALICO_K8S_NODE_REF
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              value: "1440"
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Auto-detect the BGP IP address.
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: FELIX_HEALTHENABLED
              value: "true"
            # Additional Felix configuration options
            # Enable CNX Prometheus denied packet reporting
            - name: FELIX_PROMETHEUSREPORTERENABLED
              value: "true"
            # Make CNX Prometheus metrics available on port 9081
            - name: FELIX_PROMETHEUSREPORTERPORT
              value: "9081"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            httpGet:
              path: /liveness
              port: 9099
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /readiness
              port: 9099
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - mountPath: /calico-secrets
              name: etcd-certs
        # This container installs the Calico CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: quay.io/calico/cni:v3.1.2
          command: ["/install-cni.sh"]
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
            - mountPath: /calico-secrets
              name: etcd-certs
      volumes:
        # Used by calico/node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0400

---

# This manifest deploys the Calico Kubernetes controllers.
# See https://github.com/projectcalico/kube-controllers
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      # The controllers must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      containers:
        - name: calico-kube-controllers
          image: quay.io/calico/kube-controllers:v3.1.2
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: policy,profile,workloadendpoint,node
          volumeMounts:
            # Mount in the etcd TLS secrets.
            - mountPath: /calico-secrets
              name: etcd-certs
      volumes:
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0400

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system
```