Skip to content

Instantly share code, notes, and snippets.

View yohei1126's full-sized avatar
🏠
Working from Singapore

Yohei Onishi yohei1126

🏠
Working from Singapore
76 followers · 150 following
View GitHub Profile
@yohei1126
yohei1126 / run.md
Last active July 2, 2019 09:42
Spark-submit to local k9s cluster (no VM driver) on Ubuntu

Create a GCE instance

gcloud compute instances create test-k8s \
  --boot-disk-size 30G \
  --machine-type n1-standard-4 \
  --image-project ubuntu-os-cloud \
  --image-family ubuntu-1804-lts \
  --zone us-central1-a
@yohei1126
yohei1126 / Initialize-Kubernetes-on-Ubuntsu.md
Last active July 1, 2019 12:38
Ubuntu 18.04 LTS 
$ sudo kubeadm init --pod-network-cidr=10.128.0.0/20

# disable swap https://github.com/kubernetes/kubernetes/issues/53533
$ sudo swapoff -a

$ sudo kubeadm init --pod-network-cidr=10.128.0.0/20

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
@yohei1126
yohei1126 / gist:19dd5b8e3c6609498ba1c38467e46a26
Created June 23, 2019 12:56
spark-submit failed on k8s
```
019-06-23 12:49:43 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2754)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
@yohei1126
yohei1126 / 18.1
Last active June 1, 2017 12:31
Programming in Scala 18 章 Mutable Objects ref: http://qiita.com/yohei1126@github/items/9326ab53c519690d9584
import scala.language.postfixOps
class BankAccount {
private var bal: Int = 0
def balance: Int = bal
def deposit(amount: Int) = {
require(amount > 0)
bal += amount
}
def withdraw(amount: Int): Boolean = {
if (amount > bal) false
@yohei1126
yohei1126 / Callbacks.ts
Created May 11, 2017 01:54
Typedoc sample
export type Callback = () => void;
@yohei1126
yohei1126 / file0.txt
Last active July 11, 2016 09:29
TypeScriptを使ってNode.jsとブラウザに対応したJSコード（UMD形式）を生成する ref: http://qiita.com/yohei1126@github/items/a04770e0e41c387144b6
% tree
.
├── README.md
├── index.html
├── package.json
├── src
│   ├── helloModule.ts
│   ├── index.ts
│   └── wordModule.ts
└── tsconfig.json
@yohei1126
yohei1126 / file0.txt
Last active December 20, 2015 09:48
Scala コードの列挙型と DB（Slick）の整数型 / 文字列型をマッピングする ref: http://qiita.com/yohei1126@github/items/515e8463a0457059fdce
// 文字列型 にマッピングするEnum
object Color extends Enumeration {
type Color = Value
val Blue = Value("Blue")
val Red = Value("Red")
val Green = Value("Green")
}
// 整数型にマッピングするEnum
object Bar extends Enumeration {
@yohei1126
yohei1126 / file0.txt
Last active December 23, 2015 03:33
Play 2.3.xから2.4.x、Slick 2.xから3.1.xへの移行 ref: http://qiita.com/yohei1126@github/items/d60c6e498bc57c07b880
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.4.0")
@yohei1126
yohei1126 / file0.txt
Created December 9, 2015 12:42
Jetty WebSocketClientをシングルスレッドで動かす ref: http://qiita.com/yohei1126@github/items/27b96f7e7fa2a6b13737
Executor executor = Executors.newFixedThreadPool(2);
WebSocketClient client = new WebSocketClient(executor);
@yohei1126
yohei1126 / REST_sample
Last active August 29, 2015 14:04
How to call the Automatic REST API from Android
//URL to call REST API
final HttpPost httpPostRequest = new HttpPost("https://api.parse.com/1/functions/availableCars");
//Application ID
httpPostRequest.setHeader("X-Parse-Application-Id", "App-key");
//REST API Key
httpPostRequest.setHeader("X-Parse-REST-API-Key", "REST-API-key");
// Content type must be application/json
httpPostRequest.setHeader("Content-Type", "application/json");
try {
// You have to set valid JSON data as a form data