Skip to content

Instantly share code, notes, and snippets.

@yoyama
yoyama / aws_cli_memo.md
Created August 2, 2024 02:44
aws cli memo
aws --profile <profile> --region <region> ec2 describe-instances \
    --filters "Name=tag-key,Values=Name" "Name=tag-value,Values=<name>" \
| jq '.Reservations[] | .Instances[] | .InstanceId' 
@yoyama
yoyama / Schema2CaseClass.scala
Created January 20, 2017 07:36
Generate case class from spark DataFrame/Dataset schema.
/**
* Generate Case class from DataFrame.schema
*
* val df:DataFrame = ...
*
* val s2cc = new Schema2CaseClass
* import s2cc.implicit._
*
* println(s2cc.schemaToCaseClass(df.schema, "MyClass"))
*
@yoyama
yoyama / sql_memo.md
Last active September 8, 2023 01:16
SQL memo

Generate table as SQL

Presto:

with t1 as (select sequence(1,10) as c1)
select c from t1 
CROSS JOIN UNNEST (c1) as t(c)

BigQuery:

@yoyama
yoyama / repl_ext_jar.md
Last active December 6, 2020 07:03
Scala REPL based app. fail when file name has no extension '.jar' or '.zip'

REPL based application will fail with initialization error of REPL if its jar file name has no extension '.jar' or '.zip'.

import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.shell.{ILoop, ShellConfig}

object ReplTestMain extends App {
    println("test")
    val settings = new Settings
    settings.usejavacp.value = true
@yoyama
yoyama / corenlp_test.scala
Created April 23, 2018 01:17
spar-corenlp と自前でUDFを定義したときの比較用
package test1 {
import com.databricks.spark.corenlp.functions._
import org.apache.spark.sql.{DataFrame, SparkSession}
class Test(implicit spark: SparkSession) {
def run(df: DataFrame): DataFrame = {
import spark.implicits._
val df2 = df.select(tokenize($"Text").as("word"), lemma($"Text").as("lemma"), pos($"Text").as("pos")).persist
df2.count
df2.unpersist()
@yoyama
yoyama / snake2camel.scala
Last active November 24, 2016 06:30
snake2camel
def snake2camel(s:String):String = s.toList.foldLeft(List.empty[Char]){
case ('_'::xs, c) => c.toUpper::xs
case (xs, c) => c::xs
}.reverse.mkString("")