Skip to content

Instantly share code, notes, and snippets.

@l15k4
Created December 12, 2016 14:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save l15k4/e566ffdaa45fb9681f3183ba0429c18e to your computer and use it in GitHub Desktop.
Save l15k4/e566ffdaa45fb9681f3183ba0429c18e to your computer and use it in GitHub Desktop.
val writer = new ParquetWriter[Group](outFile, new GroupWriteSupport, codec, blockSize, pageSize, dictPageSize, true, true, ParquetProperties.WriterVersion.PARQUET_2_0, configuration)
try {
records.foreach { case ClientRecord(cid, diid, gwid, dvc, geo, timestamp, kvs) =>
val group = new SimpleGroup(schema)
.append("timestamp", timestamp)
.append("cid", cid)
.append("diid", diid)
.append("gwid", gwid)
.append("dvc", dvc)
.append("geo", geo)
val kvsGroup = group.addGroup("kvs")
kvs.foreach { case (k,v) =>
kvsGroup
.addGroup("kv")
.append("key",k)
.append("value",v)
}
writer.write(group)
}
} finally writer.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment