cluster_minicube.md

How to install cluster locally

Hey, ones I fall into troble that I need to modify and test docker container which is running on our cluster. So the best I can do is to run everything locally. So let's do it. I run everything on Ubuntu 16.04.3 LTS.

What we need to install

1. Docker

It is easy. Instrucion for Ubuntu is here

2. kubectl

It is a tool for deploying and managing applications on Kubernetes Here guide I follow. We use version 1.5 now, so let's install it.

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.5.0/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

3. Minikube or Kubernetes. From doc:

Kubernetes is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling of applications.

and

Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.

Let's try with Minikube first. Install it like in docs.

1. We need a Hypervisor. Let's use VirtualBox. Here you can download package. Be careful with 32/64 bit versions (Read the note).
2. Install kubectl. It is done already.
3. Install Minikube

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube
sudo mv minikube /usr/local/bin/

Let's test it and we need to just follow Test Kubernetes Locally via Minikube article:

Try to run minikube start first. If you have any troubles, first check that your VirtualBox installation works well. Run VirtualBox and try to create a new virtual machine. Resolve any issues if you have. Also, you need to enable Intel Virtualization Technology in your BIOS.

If everything works well you see something like:

Starting local Kubernetes v1.9.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from the config file.

You can try to follow an example from Quickstart section to be 100% sure.

4. Helm

Helm is The Kubernetes Package Manager. Installation:

https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash

Now you need to initialize it. Be sure that you start your cluster: minikube start and run

helm init

BTW, to remove current cluster installation run helm delete --purge p-spark

Ok, we installed everything.

Run spark cluster

Finally. We can deploy source{d} container with a spark. Just follow the commands below, adopded from Rafa's guide:

git clone https://github.com/src-d/charts.git
cd charts/spark
helm install -n p-spark . --set \
Worker.TemporaryDir="/spark-temp-data",\
Worker.AdditionalPodContainers.bblfshd.image=bblfsh/bblfshd:v2.2.0,\
Worker.AdditionalPodContainers.bblfshd.securityContext.privileged=true,\
Worker.AdditionalPodContainers.bblfshd.lifecycle.postStart.exec.command=\{"bblfshctl","driver","install","--all"\}

if you run kubectl get pods you should get something like:

NAME                                   READY     STATUS    RESTARTS   AGE
p-spark-master-555cf4f4c-9s45r         1/1       Running   0          1m
p-spark-webui-proxy-7d89954654-g8bxn   1/1       Running   0          1m
p-spark-worker-8d9w8                   2/2       Running   0          1m

and now we should follow yet another guide from Vadim to get spark shell. You need to follow all steps from section Obtain the Spark shell

If you have any problems with something Killed unexpectedly, that means your virtual machine does not have enough resources. Run minicube stop on a host machine to shut down virtual one, run virtualbox, find minikube virtual machine and extend RAM/CPU amount in settings (Ctrl+S to open it). Run minicube start and try one time more.

Here you are! My congratulations. You can work localy.