ikoniaris

Experimentation on Spark's SaveMode

Experiment on the effect of different SaveMode and Cassandra starting from a populated table

Summary

If the cassandra table that spark targets exists then

• SaveMode.Append will update it
• SaveMode.Overwrite will truncate and insert (but it requires option "confirm.truncate" -> "true")
• SaveMode.Ignore will not perform any action on existing table
• SaveMode.ErrorIfExists (default) will throw the following exception: https://github.com/datastax/spark-cassandra-connector/blob/v2.0.6/spark-cassandra-connector/src/main/scala/org/apache/spark/sql/cassandra/DefaultSource.scala#L93-L96

ikoniaris

import java.io.InputStream

import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils
import org.apache.spark.sql.{ DataFrame, Row }

import org.postgresql.copy.CopyManager
import org.postgresql.core.BaseConnection

val jdbcUrl = s"jdbc:postgresql://..." // db credentials elided
val connectionProperties = {

ikoniaris

#!/usr/bin/python

# Converts LinkedIn connections exported as CSV to VCF (vCards), which can then be 
# imported into contact managers like Monica: https://github.com/monicahq/monica
# 
# LinkedIn instructions: https://www.linkedin.com/help/linkedin/answer/66844/exporting-connections-from-linkedin?lang=en
# vCard 3.0 information: https://en.wikipedia.org/wiki/VCard#vCard_3.0

import csv
import sys

ikoniaris

<?php
/**
 * A helper file for Laravel 5, to provide autocomplete information to your IDE
 * Generated for Laravel 5.5.13 on 2017-09-28.
 *
 * @author Barry vd. Heuvel <barryvdh@gmail.com>
 * @see https://github.com/barryvdh/laravel-ide-helper
 */
namespace  {
    exit("This file should not be included, only analyzed by your IDE");

ikoniaris

### Keybase proof

I hereby claim:

  * I am ikoniaris on github.
  * I am ikoniaris (https://keybase.io/ikoniaris) on keybase.
  * I have a public key whose fingerprint is 21E5 98B3 6313 6A44 25AF  3D39 EA07 CD0C ECC5 2F6E

To claim this, I am signing this object: