Skip to content

Instantly share code, notes, and snippets.

View minyk's full-sized avatar
😀
Containerize! all the things!

Drake Youngkun Min minyk

😀
Containerize! all the things!
66 followers · 64 following
View GitHub Profile
@minyk
minyk / spark-on-k8s.md
Last active May 11, 2022 01:26

Presentations and articles for Spark-on-K8S

For own self-education purpose

General

  1. Apache Spark on Kubernetes: A Technical Deep Dive - Yinan Li, Google, KubeCon + CloudNativeCon China 2018, Nov 15, 2018
  2. Ready to run! Get Started with Spark on Kubernetes, Jean-Yves Stephan, Spot.io, Feb 1, 2022
  3. Patterns and Anti-Patterns of running Apache bigdata projects in Kubernetes, Marton Elek, ApacheCon EU 2019, Oct 24, 2019
@minyk
minyk / spark-hadoop-aws.md
Created May 22, 2020 13:30
Apache Spark and hadoop-aws(s3a) related resources
@minyk
minyk / local-dir-hostpath.md
Last active May 21, 2020 07:28
Learning how to properly run Apache spark on the Kunernetes environments

Using spark.kubernetes.driver.volumes... configurations:

spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.path=/tmp/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.options.path=/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.path=/tmp/data2
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.options.path=/data2
@minyk
minyk / ingress.yaml
Last active October 29, 2019 05:31
NodePort configuration for K8S Dashboard
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kube-system
spec:
rules:
- host: dashboard.192.168.100.12.nip.io
http:
paths:
@minyk
minyk / gb-back.yaml
Last active August 16, 2019 09:10
k8s-devops-20190816
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: guestbook-back
spec:
replicas: 1
selector:
matchLabels:
type: app
@minyk
minyk / custom_s3_endpoint_in_spark.md
Last active December 28, 2021 08:03 — forked from tobilg/custom_s3_endpoint_in_spark.md
Description on how to use a custom S3 endpoint (like Rados Gateway for Ceph)

Custom S3 endpoints with Spark

To be able to use custom endpoints with the latest Spark distribution, one needs to add an external package (hadoop-aws). Then, custum endpoints can be configured according to docs.

Use the hadoop-aws package

bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.7.2

SparkContext configuration

@minyk
minyk / papers-for-the-container-tech.md
Last active January 26, 2024 01:10

Papers (and articles) for the Container Technologies

For own self-education purpose

Containers

  1. TBD
  2. Docker: Lightweight Linux Containers for Consistent Development and Deployment, Dirk Merkel, Linux Journal, March 2014
  3. Networking in Containers and Container Clusters, Victor Marmol, Rohit Jnagal, and Tim Hockin, netdev 0.1, Feb 14-17, 2015
@minyk
minyk / marathon.json
Created November 16, 2018 09:33
Marathon Deploy JSON with no-frontend.
{
"id": "/mw/memory-stat",
"cpus": 1,
"mem": 2048,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"forcePullImage": true,
"image": "harbor.ajway.kr/middleware/mw:1.0",
@minyk
minyk / README.md
Last active September 4, 2018 08:31
HL Fabric block data fetch with cendhu/fetch-block

build

  • Get source
$ go get -u github.com/cendhu/fetch-block
  • Get govendor
$ go get -u github.com/kardianos/govendor
  • build
@minyk
minyk / tadpole.log
Created December 1, 2017 08:40
Tadpole 1.7.5r2
tadpole_1 | 2017-12-01 08:38:59,033 ERROR - TadpoleSystem_UserQuery : saveLoginHistory save login history
tadpole_1 | com.ibatis.common.jdbc.exception.NestedSQLException:
tadpole_1 | --- The error occurred in com/hangum/tadpole/engine/query/internal/system/Tadpole-System-Common.xml.
tadpole_1 | --- The error occurred while executing update.
tadpole_1 | --- Check the INSERT INTO login_history(user_seq, login_ip, succes_yn, fail_reason) VALUES (?, ?, ?, ?); .
tadpole_1 | --- Check the SQL Statement (preparation failed).
tadpole_1 | --- Cause: java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (no such table: login_history)
tadpole_1 | at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeUpdate(MappedStatement.java:107)
tadpole_1 | at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.insert(SqlMapExecutorDelegate.java:393)
tadpole_1 | at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.insert(SqlMapSessionImpl.java:82)