Skip to content

Instantly share code, notes, and snippets.

View bigorn0's full-sized avatar
💭
Enthousiast

Ugo Bechameil bigorn0

💭
Enthousiast
10 followers · 14 following
  • UB1
  • Brussels
View GitHub Profile
@bigorn0
bigorn0 / pg-test.js
Created November 8, 2021 22:35
#!/usr/bin/env node
// Replace the
// chmod +x pg-test.js
// npm install --save pg
// ./pg-test.js
@bigorn0
bigorn0 / deepmerge.go
Created September 3, 2018 12:05
package main
import (
"encoding/json"
"fmt"
"github.com/ghodss/yaml"
"io/ioutil"
"path/filepath"
)
@bigorn0
bigorn0 / Spark apply function on multiple columns at once
Created June 6, 2017 12:00
You can use select with varargs including *:
import spark.implicits._
df.select($"*" +: Seq("A", "B", "C").map(c =>
sum(c).over(Window.partitionBy("ID").orderBy("time")).alias(s"cum$c")
): _*)
This:
Maps columns names to window expressions with Seq("A", ...).map(...)
@bigorn0
bigorn0 / factory.md
Created February 14, 2017 08:39
@bigorn0
bigorn0 / 1-restify-server-cheatsheet.js
Created November 28, 2016 21:27 — forked from LeCoupa/1-restify-server-cheatsheet.js
Restify CheatSheet. More: http://journal.gentlenode.com/node-3-restify-cheatsheet/
// Restify Server CheatSheet.
// More about the API: http://mcavage.me/node-restify/#server-api
// Install restify with npm install restify
// 1.1. Creating a Server.
// http://mcavage.me/node-restify/#Creating-a-Server
var restify = require('restify');
@bigorn0
bigorn0 / Second approach
Last active November 28, 2016 21:38
import java.sql._
dataframe.coalesce("NUMBER OF WORKERS").mapPartitions((d) => Iterator(d)).foreach { batch =>
val dbc: Connection = DriverManager.getConnection("JDBCURL")
val st: PreparedStatement = dbc.prepareStatement("YOUR PREPARED STATEMENT")
batch.grouped("# Of Rows you want per batch").foreach { session =>
session.foreach { x =>
st.setDouble(1, x.getDouble(1))
st.addBatch()
}
@bigorn0
bigorn0 / dood_cleanup.sh
Created August 25, 2016 22:25
Cleanup Dood
#!/bin/sh
str_in_list() {
str="$1"
shift
list="$@"
if test "${list#*$str}" != "$list"
then
return 0 # $str is in $list
else
@bigorn0
bigorn0 / Run_spark_submit_from_scala.md
Last active August 24, 2016 21:08
Run_spark_submit_from_scala.md

Running spark jogs from scala code

http://henningpetersen.com/post/22/running-apache-spark-jobs-from-applications

Example out of spark test classes

// NOTE: This is an expensive operation in terms of time (10 seconds+). Use sparingly.
  private def runSparkSubmit(args: Seq[String]): Unit = {
    val sparkHome = sys.props.getOrElse("spark.test.home", fail("spark.test.home is not set!"))
    val process = Utils.executeCommand(
 Seq("./bin/spark-submit") ++ args,
@bigorn0
bigorn0 / docker-security.md
Created August 24, 2016 21:05
@bigorn0
bigorn0 / spark_logging-log4j.md
Last active July 28, 2016 22:32

How to split/redirect part of the logs based on level

apache-spark-stderr-and-stdout

Try this in log4j.properties passed to Spark (or modify default configuration under Spark/conf)

# Log to stdout and stderr
log4j.rootLogger=INFO, stdout, stderr