Skip to content

Instantly share code, notes, and snippets.

@bepcyc

bepcyc/startingOffsets.scala

Created October 11, 2018 14:35
Show Gist options
  • Save bepcyc/b75c3f0e6531f8af27e163261482ee7b to your computer and use it in GitHub Desktop.
Save bepcyc/b75c3f0e6531f8af27e163261482ee7b to your computer and use it in GitHub Desktop.
Download ZIP
form starting offsets string for spark
Raw
startingOffsets.scala
"""{"acp_prod.devices": {""" + df.select($"partition", $"offset").groupBy($"partition").agg(max($"offset")).as[(Int, Long)].collect.map{case (p, o) => s""""$p": $o"""}.mkString(",") + "}}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment