etheleon

There's multiple ways to generate a kfp component:

1. from python function
2. from file/text

From file/text

You'll use component.load_component_from_<file|text> when you need to interface with a command line tool.

etheleon

Introduction

If you would like to work in the same environment as jupyterlab, check out Theia, the cloud IDE. The following highlights how we can install theia in your jupyter notebook's container.

For more information here's a post describing how Google Cloud Platform is using Theia as their default cloud IDE: https://cloud.google.com/blog/products/application-development/introducing-cloud-shell-editor.

• Usage

etheleon

Common tasks

Presto	Spark
array_join(array[year, month, day], '-', 'NA')	CONCAT_WS('-', col1, col2, col3)
DATE_ADD('day', -7, date '2021-07-01')	date_sub('2021-07-01', 7)
array_join('one', 'two', 'three')[1] # one indexed	array_join('one', 'two', 'three')[0] # zero indexed

Partition Naming convention

etheleon

Class

class Upper {
  def upper(strings: String*): Seq[String] = {
    strings.map((s:String) => s.toupperCase())
  }
}

etheleon

-- ALTER TABLE schema.table DROP IF EXISTS PARTITION (year='2021', month='01', day='11', hour='01')
ALTER TABLE pricing.demand_tbl ADD
PARTITION (year='2021', month='01', day='11', hour='01') 
LOCATION 's3://datascience-bucket/wesley.goi/data/pricing/demand_tbl/year=2021/month=01/day=11/hour=01'

etheleon

Introduction

Installing kubeflow on localmachine is not a simple task. Documentation on the official website might be outdated. At the time of writing, the solutions suggested include miniKF and microk8s. The later sets up GPU passthrough effortlessly.

• Install microk8s
• Enable microk8s features

etheleon

There's no need to create a heartbeat endpoint, with gRPC you could just use the grpc-health-probe

You can use either of two tools grpcurl or the web equivalent grpcUI (similar to postman but you cannot store the collection, but you can keep the request data as a .json file

Installation

grpcURL

etheleon

Use the nbconvert / jupyter-nbconvert CLI to converting IPYNB to HTML

$ NOTEBOOK=file_name.ipynb
$ jupyter-nbconvert --to html $NOTEBOOK

You can add tags to hide cells, eg. code or outputs which you do not want in shown in the HTML

etheleon

You got your machine, now you want to start working immediately on your 1st day at work. What do you do?

1. Install iterm2

curl -O https://iterm2.com/downloads/stable/iTerm2-3_3_12.zip

2. Install brew (and xcode)

etheleon

Step 1: Install gcsFUSE (linux)

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get install gcsfuse