Skip to content

Instantly share code, notes, and snippets.

View itayB's full-sized avatar
:octocat:

Itay itayB

:octocat:
30 followers · 18 following
View GitHub Profile
@itayB
itayB / airflow-staging-2.2.5-without-data.sql
Created August 23, 2022 10:36
Our MySql dump without data
-- MySQL dump 10.13 Distrib 8.0.29, for macos12.2 (x86_64)
--
-- Host: airflow-ml-XXXXX.dynamicyield.com Database: airflow
-- ------------------------------------------------------
-- Server version 8.0.20
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!50503 SET NAMES utf8mb4 */;
@itayB
itayB / airflow_audience_coverage.json
Last active April 16, 2024 09:10
{"schemaVersion":1,"label":"coverage","message":"71.48%","color":"green"}
@itayB
itayB / jupyternotebook.Dockerfile
Created March 7, 2021 17:24
Dockerfile for jupyter notebook + Spark
FROM itayb/spark:3.1.1-hadoop-3.2.0-aws
RUN pip install \
notebook==6.2.0 \
ipynb==0.5.1 \
sparkmonitor==1.1.1 \
pyspark==3.1.1
# install extension to monitor spark
RUN jupyter nbextension install sparkmonitor --py --user --symlink
@itayB
itayB / spark.Dockerfile
Created March 7, 2021 17:15
Spark with AWS support
FROM itayb/spark:3.1.1-hadoop-3.2.0
RUN apt-get update && apt install -y \
wget \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /var/cache/apt/*
RUN cd /opt/spark/jars \
&& wget "https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.888/aws-java-sdk-bundle-1.11.888.jar"
RUN cd /opt/spark/jars \
@itayB
itayB / app1.ipynb
Created March 6, 2021 20:03
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@itayB
itayB / spark_application.ipynb
Last active March 7, 2021 20:19
Spark session creation
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@itayB
itayB / stocks.csv
Created March 6, 2021 19:45
InvoiceNo StockCode Description Quantity InvoiceDate UnitPrice CustomerID Country
536365 85123A WHITE HANGING HEART T-LIGHT HOLDER 6 2010-12-01 08:26:00 2.55 17850.0 United Kingdom
536365 71053 WHITE METAL LANTERN 6 2010-12-01 08:26:00 3.39 17850.0 United Kingdom
536365 84406B CREAM CUPID HEARTS COAT HANGER 8 2010-12-01 08:26:00 2.75 17850.0 United Kingdom
@itayB
itayB / localstack.yaml
Last active March 9, 2021 08:53
AWS locally
apiVersion: apps/v1
kind: Deployment
metadata:
name: localstack
labels:
release: localstack
spec:
replicas: 1
strategy:
type: RollingUpdate
@itayB
itayB / jupyter.yaml
Last active March 8, 2021 16:31
Jupyter notebook with Spark 3.1.1
apiVersion: v1
kind: ServiceAccount
metadata:
name: jupyter
labels:
release: jupyter
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
@itayB
itayB / running k8s on mac
Last active January 25, 2020 17:08
# use kubernetes with 'minikube' (can be set from the docker ui in mac)
# start the dashboard (web UI)
minikube dashboard
# the command above should open Chrome browswer autmatically to some like:
http://127.0.0.1:56176/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/overview?namespace=default
FAQ