How to run Cloudflow in minikube

cloudflowminikube.md

After installing minikube via brew you can start it as follows. Assumes you have Mac for Desktop running but minikube supports other virtualization options. Make sure you give Docker a significant amount of drivespace (100Gb) and as much memory and cpus and as you provide to minikube.

Requirements: minikube, kubectl and helm should be installed

minikube start --cpus 4 --memory 16000 --kubernetes-version=1.15.6

Set up the local environment for kubectl commands.

eval $(minikube -p minikube docker-env)

Install the NFS file system (optional, you can use the default minikube file system)

helm repo list

If you don't see https://kubernetes-charts.storage.googleapis.com then add it...

helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update

Install nfs

./test/install-nfs.sh

Download the latest install script linked to from the Cloudflow release page

https://github.com/lightbend/cloudflow/releases/tag/v2.0.7

Run the install script. Note we cannot use default bash, it needs version 4 or above. You can install bash via brew which will be version 5 (at time of writing).

Download the install script and execute it. You can choose nfs for the storage options. Don't forget to change the path below to match your bash install and the version of Cloudflow.

brew install bash
/usr/local/Cellar/bash/5.0.18/bin/bash ~/Downloads/bootstrap-install-script-2.0.5.sh

If everything goes ok you will see that cloudflow is installed and can start using Cloudflow commands

kubectl cloudflow list

Let's deploy the taxi example

First we need to reduce the resources the example uses so that it will work on our local machine. Make a file called minikube.conf and include the following contents, which reduces the required resources to run.

cloudflow.streamlets {
  processor  {
    kubernetes.pods.pod.containers.container {
      resources {
        requests {
          cpu = "0.1"
          memory = "256M"
        }
        limits {
          memory = "800M"
        }
      }
    }
  }
  generator  {
    kubernetes.pods.pod.containers.container {
      resources {
        requests {
          cpu = "0.1"
          memory = "256M"
        }
        limits {
          memory = "800M"
        }
      }
    }
  }
  logger  {
    kubernetes.pods.pod.containers.container {
      resources {
        requests {
          cpu = "0.1"
          memory = "256M"
        }
        limits {
          memory = "800M"
        }
      }
    }
  }
}

Now you can deploy the taxi example by sbt buildAll then cd cloudflow/examples/taxi-ride and run with the config we created and scale the processors down to 1 (the default is 3)

kubectl cloudflow deploy target/taxi-ride-fare.json --conf minikube.conf --scale processor=1

Now you can use kubectl get pods --all-namespaces to keep an eye on the Cloudflow and Cloudflow sample progress; eventually all the pods should be running and you can check the logs to see the application running.