Skip to content

Instantly share code, notes, and snippets.

View smiklos's full-sized avatar
:octocat:
cooling

Miklos Szots smiklos

:octocat:
cooling
6 followers · 1 following
View GitHub Profile
@smiklos
smiklos / wait-for-local-port.sh
Created January 21, 2019 14:29
wait_for_local_port() {
local port=$1
local timeout=$2
local count=0
local ret=1
echo "waiting for S3:$port"
while [[ "$ret" -eq "1" && "$count" -lt "$timeout" ]] ; do
nc -z -w 1 localhost $port
ret=$?
if [ ! "$ret" -eq "0" ]; then
@smiklos
smiklos / numericColumns.scala
Created June 21, 2018 19:01
private[sql] def numericColumns: Seq[Expression] = {
schema.fields.filter(_.dataType.isInstanceOf[NumericType]).map { n =>
queryExecution.analyzed.resolveQuoted(n.name, sparkSession.sessionState.analyzer.resolver).get
}
}
@smiklos
smiklos / dataframe-aggs.scala
Created June 21, 2018 18:51
val df = spark.createDataFrame(Seq(("col1", "col2", 4, 5, 7, 5),
("col1", "col2", 2, 0, 2, 2),
("col1", "col2", 2, 0, 2, 2),
("col1", "col1", 2, 0, 2, 2),
("col1", "col1", 5, 10, 3, 4)))
.toDF("first_group", "second_group", "col1", "col2", "col3", "col4")
df.groupBy("first_group", "second_group").min()
.groupBy("first_group").avg()
@smiklos
smiklos / IntelliJ_IDEA__Perf_Tuning.txt
Created June 1, 2018 12:32 — forked from P7h/IntelliJ_IDEA__Perf_Tuning.txt
Performance tuning parameters for IntelliJ IDEA. Add these params in idea64.exe.vmoptions or idea.exe.vmoptions file in IntelliJ IDEA. If you are using JDK 8.x, please knock off PermSize and MaxPermSize parameters from the tuning configuration.
-server
-Xms2048m
-Xmx2048m
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:+UseParNewGC
-XX:ParallelGCThreads=4
-XX:MaxTenuringThreshold=1
@smiklos
smiklos / keybase.md
Created October 12, 2017 08:03

Keybase proof

I hereby claim:

  • I am smiklos on github.
  • I am sm1kl0s (https://keybase.io/sm1kl0s) on keybase.
  • I have a public key ASDTE5xqWekSfNLriYPFhzpJJNM-cVmkd_q21FeTnAnDjQo

To claim this, I am signing this object: