Skip to content

Instantly share code, notes, and snippets.

View erikerlandson's full-sized avatar

Erik Erlandson erikerlandson

142 followers · 2 following
View GitHub Profile
@erikerlandson
erikerlandson / s2i-run.sh
Last active December 1, 2020 18:06
Example of S2I from jupyter notebook to a batch spark job (source to image)
#!/bin/bash
export NOTEBOOK_SCRIPT=$(basename patched-${S2I_SOURCE_NOTEBOOK} .ipynb).ipy
export S2I_SPARK_EXECUTOR_MEMORY=${S2I_SPARK_EXECUTOR_MEMORY:-"3900m"}
export S2I_SPARK_EXECUTORS=${S2I_SPARK_EXECUTORS:-"2"}
export S2I_SPARK_DRIVER_MEMORY=${S2I_SPARK_DRIVER_MEMORY:-"4g"}
export S2I_EMAIL_FROM=${S2I_EMAIL_FROM:-noreply@some.email.com}
export S2I_SMTP_SERVER=${S2I_SMTP_SERVER:-smtp.server.com:25}
@erikerlandson
erikerlandson / fragments.yaml
Created November 5, 2020 19:26
pod hax
labels:
team: opendatahub
spec:
podMetricsEndpoints:
- interval: 30s
selector:
matchLabels:
team: opendatahub
spec:
podMonitorSelector:
@erikerlandson
erikerlandson / gist:56ee23ae2a5a89cf3c7d551ffcf6e7eb
Created October 15, 2020 18:51
notes
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.1
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
RUN microdnf install nodejs npm nodejs-nodemon nss_wrapper \
&& microdnf update \
&& microdnf clean all \
&& mkdir /deployments \
&& chown 1001 /deployments \
&& chmod "g+rwX" /deployments \
&& chown 1001:root /deployments
@erikerlandson
erikerlandson / sql-alchemy-example.py
Created May 4, 2020 22:28
writing pandas data into postgresql using sqlalchemy
import pandas as pd
data = pd.DataFrame(list(range(100)), columns=["data"])
pip install psycopg2
import sqlalchemy
from sqlalchemy import create_engine
import psycopg2
@erikerlandson
erikerlandson / spark-jdbc-connect-structures.py
Created May 1, 2020 00:12
spark JDBC connect strings and structures
def spark_jdbc_connect_structures( \
sqltype, \
host, \
db, \
user, \
password, \
driver = None, \
port = None \
):
sqltype = sqltype.lower()
@erikerlandson
erikerlandson / cast-literal-type-to-double.scala
Created April 16, 2020 22:49
casting numeric literal types at comile time
// XDouble, ToDouble, OpAuxDouble from singleton-ops
// Greater from refined
implicit def impliesGTGT[L, R, LD <: XDouble, RD <: XDouble](implicit
ld: OpAuxDouble[ToDouble[L], LD],
rd: OpAuxDouble[ToDouble[R], RD],
q: OpAuxBoolean[LD >= RD, true]): Greater[L] ==> Greater[R] =
new Implies[Greater[L], Greater[R]] {}
@erikerlandson
erikerlandson / bind-literal-type.scala
Created April 16, 2020 22:47
scala> def foo[T <: Singleton](ce: T)(implicit tt: WeakTypeTag[T]): String = tt.tpe.toString
foo: [T <: Singleton](ce: T)(implicit tt: reflect.runtime.universe.WeakTypeTag[T])String
scala> foo(5)
res37: String = Int(5)
scala> foo("abc")
res38: String = String("abc")
scala> val x = 5
@erikerlandson
erikerlandson / gist:f042fffde7e2aeb5f010e0254b873971
Created March 31, 2020 23:32
wrap a .ipy file in a big try/catch to catch any exception
$ echo "try:" > eje.ipy
$ awk '{print " ", $0}' < simple.ipy >> eje.ipy
$ cat >> eje.ipy <<- EOF
> except Exception as e:
> print("===========")
> print(e)
> print("===========")
> EOF
$ ~/conda/bin/ipython eje.ipy
45
@erikerlandson
erikerlandson / install-odh.md
Last active August 11, 2020 20:25
open data hub manual install examples (ODH circa 0.5.1) 
oc apply -f deploy/crds/opendatahub_v1alpha1_opendatahub_crd.yaml
oc apply -f deploy/service_account.yaml
oc apply -f deploy/role.yaml
oc apply -f deploy/role_binding.yaml
oc apply -f deploy/operator.yaml
# customize odh manifest
oc apply -f my_environment_cr.yaml
@erikerlandson
erikerlandson / spark-jdbc-examples.py
Last active November 13, 2019 23:50
Accessing SQL databases via JDBC, using spark, from a cloud-resident jupyter notebook
# SQL JDBC drivers can be downloaded from maven coords, if they are exposed that way.
# Otherwise they may be installed as naked jars and added to the path
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = \
# any other pyspark-shell args as well ... \
"""--packages 'org.postgresql:postgresql:42.2.8,com.ibm.db2.jcc:db2jcc:db2jcc4' pyspark-shell"""
# attach to the spark cluster
spark = SparkSession.builder.master("spark://spark-cluster-poweruser:7077").getOrCreate()