Skip to content

Instantly share code, notes, and snippets.

View minyk's full-sized avatar
😀
Containerize! all the things!

Drake Youngkun Min minyk

😀
Containerize! all the things!
65 followers · 64 following
View GitHub Profile
@minyk
minyk / papers-for-the-container-tech.md
Last active January 26, 2024 01:10

Papers (and articles) for the Container Technologies

For own self-education purpose

Containers

  1. TBD
  2. Docker: Lightweight Linux Containers for Consistent Development and Deployment, Dirk Merkel, Linux Journal, March 2014
  3. Networking in Containers and Container Clusters, Victor Marmol, Rohit Jnagal, and Tim Hockin, netdev 0.1, Feb 14-17, 2015
@minyk
minyk / spark-on-k8s.md
Last active May 11, 2022 01:26

Presentations and articles for Spark-on-K8S

For own self-education purpose

General

  1. Apache Spark on Kubernetes: A Technical Deep Dive - Yinan Li, Google, KubeCon + CloudNativeCon China 2018, Nov 15, 2018
  2. Ready to run! Get Started with Spark on Kubernetes, Jean-Yves Stephan, Spot.io, Feb 1, 2022
  3. Patterns and Anti-Patterns of running Apache bigdata projects in Kubernetes, Marton Elek, ApacheCon EU 2019, Oct 24, 2019
@minyk
minyk / custom_s3_endpoint_in_spark.md
Last active December 28, 2021 08:03 — forked from tobilg/custom_s3_endpoint_in_spark.md
Description on how to use a custom S3 endpoint (like Rados Gateway for Ceph)

Custom S3 endpoints with Spark

To be able to use custom endpoints with the latest Spark distribution, one needs to add an external package (hadoop-aws). Then, custum endpoints can be configured according to docs.

Use the hadoop-aws package

bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.7.2

SparkContext configuration

@minyk
minyk / si7021.py
Created November 8, 2016 04:49
Si7021 Sensor Module for Micropython-ESP8266
from time import sleep_ms
from machine import Pin, I2C
# Default Address
SI7021_I2C_DEFAULT_ADDR = 0x40
# Commands
CMD_MEASURE_RELATIVE_HUMIDITY_HOLD_MASTER_MODE = 0xE5
CMD_MEASURE_RELATIVE_HUMIDITY = 0xF5
CMD_MEASURE_TEMPERATURE_HOLD_MASTER_MODE = 0xE3
@minyk
minyk / spark-hadoop-aws.md
Created May 22, 2020 13:30
Apache Spark and hadoop-aws(s3a) related resources
@minyk
minyk / local-dir-hostpath.md
Last active May 21, 2020 07:28
Learning how to properly run Apache spark on the Kunernetes environments

Using spark.kubernetes.driver.volumes... configurations:

spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.path=/tmp/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.options.path=/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.path=/tmp/data2
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.options.path=/data2
@minyk
minyk / ingress.yaml
Last active October 29, 2019 05:31
NodePort configuration for K8S Dashboard
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kube-system
spec:
rules:
- host: dashboard.192.168.100.12.nip.io
http:
paths:
@minyk
minyk / gb-back.yaml
Last active August 16, 2019 09:10
k8s-devops-20190816
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: guestbook-back
spec:
replicas: 1
selector:
matchLabels:
type: app
@minyk
minyk / marathon.json
Created November 16, 2018 09:33
Marathon Deploy JSON with no-frontend.
{
"id": "/mw/memory-stat",
"cpus": 1,
"mem": 2048,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"forcePullImage": true,
"image": "harbor.ajway.kr/middleware/mw:1.0",
@minyk
minyk / README.md
Last active September 4, 2018 08:31
HL Fabric block data fetch with cendhu/fetch-block

build

  • Get source
$ go get -u github.com/cendhu/fetch-block
  • Get govendor
$ go get -u github.com/kardianos/govendor
  • build