Skip to content

Instantly share code, notes, and snippets.

@longcao
longcao / SparkCopyPostgres.scala
Last active Jul 6, 2020
COPY Spark DataFrame rows to PostgreSQL (via JDBC)
View SparkCopyPostgres.scala
import java.io.InputStream
import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
import org.apache.spark.sql.{ DataFrame, Row }
import org.postgresql.copy.CopyManager
import org.postgresql.core.BaseConnection
val jdbcUrl = s"jdbc:postgresql://..." // db credentials elided
val connectionProperties = {
View words.scala
/**
* Intersperses words with spaces and trims ends, accounts for basic smilies.
*/
def wordsToSentence(words: Vector[String]): String = {
val punctuation = Set(":", ";", ".", ",", "!", "?")
val smilies = Set(":)", ":(", ":<", ":}", ":|", ":o")
// @TODO really bad implementation probably, clean this up later
words.zipWithIndex.foldLeft("") { case (acc, (word, i)) =>
val isLastWordPairSmilie = if (words.isDefinedAt(i - 1)) {
@longcao
longcao / problems.md
Created Dec 20, 2015
sample scala problems to cover in cats docs?
View problems.md
  1. Handling stacked monads and the like e.g. List[Option[T]], `Future[Option[T]``
  2. Referentially transparent error handling
  3. Error accumulation/'parallel' validation
  4. Adding together Option[T]s (a la http://stackoverflow.com/a/16319667, not sure how to word this well)
  5. More explicitly typesafe collections (e.g. non-empty, guarantee that list.head never throws, etc)
@longcao
longcao / Server.scala
Created Dec 20, 2013
Server source code.
View Server.scala
import akka.actor._
object Server extends App {
val system = ActorSystem("serverSystem")
val serverActor = system.actorOf(Props(new ServerActor), name = "ServerActor")
}
class ServerActor extends Actor {
def receive = {
case message: String =>
@longcao
longcao / Client.scala
Last active Dec 31, 2015
Client source code.
View Client.scala
import akka.actor._
object Client extends App {
val system = ActorSystem("clientSystem")
val clientActor = system.actorOf(Props(new ClientActor), name = "ClientActor")
clientActor ! SayHello
}
class ClientActor extends Actor {
@longcao
longcao / client.conf
Last active Dec 31, 2015
client and server conf
View client.conf
# 'client' conf
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "127.0.0.1"
port = 2553
View dependencies.sbt
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.2.3",
"com.typesafe.akka" %% "akka-remote" % "2.2.3"
)
You can’t perform that action at this time.