smiklos

Keybase proof

I hereby claim:

• I am smiklos on github.
• I am sm1kl0s (https://keybase.io/sm1kl0s) on keybase.
• I have a public key ASDTE5xqWekSfNLriYPFhzpJJNM-cVmkd_q21FeTnAnDjQo

To claim this, I am signing this object:

smiklos

-server
-Xms2048m
-Xmx2048m
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:PermSize=512m
-XX:MaxPermSize=512m
-XX:+UseParNewGC
-XX:ParallelGCThreads=4
-XX:MaxTenuringThreshold=1

smiklos

val df = spark.createDataFrame(Seq(("col1", "col2", 4, 5, 7, 5), 
                                   ("col1", "col2", 2, 0, 2, 2), 
                                   ("col1", "col2", 2, 0, 2, 2), 
                                   ("col1", "col1", 2, 0, 2, 2), 
                                   ("col1", "col1", 5, 10, 3, 4)))
.toDF("first_group", "second_group", "col1", "col2", "col3", "col4")

df.groupBy("first_group", "second_group").min()
  .groupBy("first_group").avg()

smiklos

  private[sql] def numericColumns: Seq[Expression] = {
    schema.fields.filter(_.dataType.isInstanceOf[NumericType]).map { n =>
      queryExecution.analyzed.resolveQuoted(n.name, sparkSession.sessionState.analyzer.resolver).get
    }
  }

smiklos

wait_for_local_port() {
    local port=$1
    local timeout=$2
    local count=0
    local ret=1
    echo "waiting for S3:$port"
    while [[ "$ret" -eq "1" && "$count" -lt "$timeout" ]] ; do
        nc -z -w 1 localhost $port
        ret=$?
        if [ ! "$ret" -eq "0" ]; then