Skip to content

Instantly share code, notes, and snippets.

View rcmiii's full-sized avatar

Romaine C Martin rcmiii

4 followers · 0 following
  • Grovo
  • New York, New York
View GitHub Profile
@rcmiii
rcmiii / TypedSpark.scala
Created June 15, 2021 13:20
Typesafe Joins in Spark 2.4
import org.apache.spark.sql.{Dataset, Encoder, Encoders, SparkSession}
import scala.reflect.ClassTag
import scala.reflect.runtime.universe.TypeTag
implicit class TypedDataset[T: TypeTag: Encoder](ds: Dataset[T]) {
val spark: SparkSession = ds.sparkSession
import spark.implicits._
@rcmiii
rcmiii / generate_reccs.scala
Created May 23, 2017 19:55
spark-shell --jars ~/dev/hadoop-aws-2.7.0.jar --conf 'spark.driver.extraJavaOptions=-Ddata-warehouse-url-read=s3n://grovo-data-warehouse-dev'
import com.grovo.data.common.dao.CsvReportDao
import com.grovo.data.common.domain.TimePeriod
import com.grovo.data.cubes.enterprise.dao.EnterpriseAggregateDao
import com.grovo.data.snapshots.enterprise.dao.EnterpriseDataSnapshotsDao
import org.apache.spark.sql.{SaveMode, SparkSession}
import com.grovo.data.common.etl.SnapshotBuilder
import com.grovo.data.common.util.Config.SNAPSHOT_TASKS_PARALLELISM
import com.grovo.data.common.util.DateConversions.processingHours
@rcmiii
rcmiii / keybase.md
Created February 13, 2017 17:29
keybase.md

Keybase proof

I hereby claim:

  • I am rcmiii on github.
  • I am rcmiii (https://keybase.io/rcmiii) on keybase.
  • I have a public key ASD9CNsV64y5o71VY46fEgywGGcCUbYaHU7wZeSaagsWYQo

To claim this, I am signing this object: