hsinhoyeh

apt-get update
# do not instal gtk related package
apt-get install git cmake gcc g++ python3-dev python3-numpy libpng-dev libjpeg-dev libopenexr-dev libtiff-dev libwebp-dev

git clone https://github.com/opencv/opencv.git
cd opencv && mkdie build && cd build && cmake -D PYTHON3_PACKAGES_PATH=/usr/lib/python3/dist-packages ../

hsinhoyeh

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: gpu-operator
  name: gpu-operator
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:

hsinhoyeh

===install dockerd on ubuntu 2004===

apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
 $(lsb_release -cs) \

hsinhoyeh

# background: when you run python code with command mode (aka -c ), inspect module can't work well as it didn't 
# serialize commands into .py file. instead we can leverage dill to handle marshal or unmarshal functions for us.
# in this example, we wrap it into a python declarator.

# pip install dill

def inspectMe(f):
    import dill as pickle
    import base64
    def wrapper(*arg, **kwargs):

hsinhoyeh

this is on-prem solution to run minikube(v1.10.x) + kubeflow(v1.0.x)+ nvidia(K80) on top of ubuntu 18.04LTE
as we want to mini cost spent on GCP (which we have $300 free credits) and launch a real k8s cluster is way of expensive.

our hardware configuration is below:
```
4 cpu + 32 G memory
200G harddisk
1 nvidia K80 GPU
```

hsinhoyeh

// ref: https://minikube.sigs.k8s.io/docs/tutorials/nvidia_gpu/
// os: ubuntu 18.04 LTE
// hardware: GCE instance with GPU(nvidia-tesla0k80)


1. install nvidia driver
```
https://gist.github.com/hsinhoyeh/495752aaf252bebdd2f3b51011dc060f
```

hsinhoyeh

// check the following doc to installation
// https://cloud.google.com/compute/docs/gpus/install-drivers-gpu

1. install gce nvidia driver:
```
curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub

```

hsinhoyeh

// previously, please refer to: https://gist.github.com/hsinhoyeh/c5f60b4cbe41a1e6478ae5ea10f47497
// ref: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/ 
// modification:
// 1. remove istio related as we already have one during the previous step
// 2. change clusteradmin to admin@kubeflow.org instead of using anonymous

cat kfctl_istio_dex.v1.0.2.yaml
```
apiVersion: kfdef.apps.kubeflow.org/v1
kind: KfDef

hsinhoyeh

//installation doc: https://www.kubeflow.org/docs/started/workstation/minikube-linux/
// note: use driver=none to disable any virtual machine on top of ubuntu

0. install docker

```
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \

hsinhoyeh

import parquet
// pip3 install parquet

import json
import glob

def parquet2json(filename):
    print(filename)
    with open(filename, "rb") as fo:
        output = "{}.json".format(filename)