abhisheyke

Keybase proof

I hereby claim:

• I am abhisheyke on github.
• I am abhisheyke (https://keybase.io/abhisheyke) on keybase.
• I have a public key ASDFxjL573smzRb4DOL1xmk3zsydjTkVgIRG1KxTs3V1vgo

To claim this, I am signing this object:

abhisheyke

+ '[' -n '' ']'
++ id -u
+ myuid=0
++ id -g
+ mygid=0
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_K8S_CMD=driver
+ echo 'Request is for driver'

abhisheyke

// Brief version of code...
 val netflow = spark
      .readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", bootstrapServers)  // kafka:9092
      .option("value.deserializer", classOf[StringDeserializer].getCanonicalName)
      .option("startingOffsets", "latest")
      .option("failOnDataLoss", false)
      .option(subscribeType, "topic1")
      .load().select(from_json(functions.col("value").cast("string"), NetflowStructType.netflowMsg, jsonOptions).alias("nftable"))

abhisheyke

Job run on two executor pods. Both execuotr pods has similar statistics.  
Details statistics captured from one of two executors.

File descriptor...
# lsof -p 14 | tail -10
java     14 root *621u  a_inode               0,10        0      8838 [eventpoll]
java     14 root *622r     FIFO                0,9      0t0 250666725 pipe
java     14 root *623w     FIFO                0,9      0t0 250666725 pipe
java     14 root *624u  a_inode               0,10        0      8838 [eventpoll]
java     14 root *625r     FIFO                0,9      0t0 250505447 pipe