Skip to content

Instantly share code, notes, and snippets.

@icexelloss

icexelloss/flint1.py

Last active January 2, 2021 04:39
Show Gist options
  • Save icexelloss/46224613d7c9fae991a4a180c13a8de4 to your computer and use it in GitHub Desktop.
Save icexelloss/46224613d7c9fae991a4a180c13a8de4 to your computer and use it in GitHub Desktop.
Download ZIP
Flint1
Raw
flint1.py
from ts.flint import FlintContext
flintContext = FlintContext(sqlContext)
df = spark.createDataFrame(
[('2018-08-20', 1.0), ('2018-08-21', 2.0), ('2018-08-24', 3.0)],
['time', 'v']
).withColumn('time', from_utc_timestamp(col('time'), 'UTC'))
# Convert to Flint DataFrame
flint_df = flintContext.read.dataframe(df)
# Use Spark DataFrame functionality
flint_df = flint_df.withColumn('v', flint_df['v'] + 1)
# Use Flint functionality
flint_df = flint_df.summarizeCycles(summarizers.count())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment