Skip to content

Instantly share code, notes, and snippets.

@burythehammer
Created March 30, 2017 15:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save burythehammer/b9c8fcd3c63bba78ca70cc49e1f0263e to your computer and use it in GitHub Desktop.
Save burythehammer/b9c8fcd3c63bba78ca70cc49e1f0263e to your computer and use it in GitHub Desktop.
Unstructured Cassandra and Spark logic
/* 1*/ val includedStatuses = Set("COMPLETED", "REPAID")
/* 2*/ val now = new Date();
/* 3*/ sc.cassandraTable("cc", "cc_transactions")
/* 4*/ .select("customerid", "amount", "card", "status", "id")
/* 5*/ .where("id < minTimeuuid(?)", now)
/* 6*/ .filter(includedStatuses contains _.getString("status"))
/* 7*/ .keyBy(row => (row.getString("customerid"), row.getString("card")))
/* 8*/ .map { case (key, value) => (key, value.getInt("amount")) }
/* 9*/ .reduceByKey(_ + _)
/*10*/ .map { case ((customerid, card), balance) => (customerid, card, balance, now) }
/*11*/ .saveToCassandra("cc", "cc_balance", SomeColumns("customerid", "card", "balance", "updated_at"))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment