benjamintanweihao

Supporting Arbitary ML Models with MlFlow

Using PyCaret as an Example

MlFlow is awesome. We use it all the time to track our ML models and their artifacts. Logging model training runs are super easy, and if you need any custom logic, the API is also pretty easy to use.

The one small caveat to this is that the model should already be supported by MlFlow. See the following list for all the models that MlFlow natively supports. However, what happens when you have to log a model that isn't natively supported by MlFlow?

Enter mlflow.pyfunc.PythonModel

benjamintanweihao

Install K3s with GPU Support

Typically installing K3s is a one-liner. However, because we need to enable Nvidia GPU support, we'll need a few extra steps.

First, run the following the download and install K3s. INSTALL_K3S_SKIP_START skips automatically starting K3s because we'll need to modify containerd to use the nvidia-container-runtime (more on that later). INSTALL_K3S_EXEC="server --no-deploy traefik" is to skip Traefik because KubeFlow uses Istio. If this were used, the Istio wouldn't initialise properly:

curl -sfL https://get.k3s.io |  INSTALL_K3S_SKIP_START=true INSTALL_K3S_EXEC="server --no-deploy traefik" sh

benjamintanweihao

; Ender 3 Custom Start G-code
M117 Getting the bed up to temp!
M140 S{material_bed_temperature_layer_0} ; Set Heat Bed temperature
M190 S{material_bed_temperature_layer_0} ; Wait for Heat Bed temperature
M117 Pre-heating the extruder!
M104 S160; start warming extruder to 160
G28 ; Home all axes
M117 Auto bed-level GO!
G29 ; Auto bed-level (BL-Touch)
G92 E0 ; Reset Extruder

benjamintanweihao

#include "FastLED.h"
#include "limits.h"

#define NUM_LEDS 60 // How many leds in your strip?
#define DATA_PIN 5 // led data transfer
#define MICROPHONE_PIN A0


long sensorMax = 0;
long sensorMin = LONG_MAX;

benjamintanweihao

; Ender 3 Custom Start G-code
M117 Getting the bed up to temp!
M140 S{material_bed_temperature_layer_0} ; Set Heat Bed temperature
M190 S{material_bed_temperature_layer_0} ; Wait for Heat Bed temperature
M117 Pre-heating the extruder!
M104 S160; start warming extruder to 160
G28 ; Home all axes
M117 Auto bed-level GO!
G29 ; Auto bed-level (BL-Touch)
G92 E0 ; Reset Extruder

benjamintanweihao

import cv2


def equalize_histogram(path):
    # Color
    print('Processing (Color): {}'.format(path))
    img = cv2.imread(path)
    lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
    lab_planes = cv2.split(lab)

benjamintanweihao

# RenderDoc Python scripts, powered by IronPython 2.7.4.1000
# The 'renderdoc' object is the Core class instance.

config = {}
config['py_lib_dir'] = 'C:\\Python27\\Lib\\'
config['save_dir'] = 'C:\\workspace\\playing-for-data\\data\\processed\\'  # where we store extraction results
# creates a prefixes for files and directories from logfilename
config['dir_prefix'] = lambda logFilename: ''
config['file_prefix'] = lambda logFilename: basename(logFilename)[:-4] + '_'

benjamintanweihao

import numpy as np
import matplotlib.pyplot as plt

filename = 'C:\\Users\\Benjamin Tan\\Desktop\\00016351_object_id.raw'

vals = np.linspace(0,1,256)
np.random.shuffle(vals)
cmap = plt.cm.colors.ListedColormap(plt.cm.jet(vals))

# Load the object_id.raw file and skip the first 4 bytes

benjamintanweihao

import tensorflow as tf
import os

from tensorflow.python.keras import Input, Model
from tensorflow.python.keras.models import load_model

from darknet import darknet_base
from definitions import ROOT_DIR
from yolo_layer import YOLOLayer

benjamintanweihao

name: kaggle
channels:
  - defaults
dependencies:
  - backcall=0.1.0=py36_0
  - bleach=2.1.3=py36_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - dbus=1.13.2=h714fa37_1
  - decorator=4.3.0=py36_0