pacman -S <package> # Install a package
pacman -Sy # Update package list
pacman -Su # Update installed packages
pacman -Ss <query> # Search packages
pacman -R <package> # Remove a package
pacman -Rs <package> # Remove a package and it's unneeded dependencies
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.spark._ | |
import org.apache.spark.streaming._ | |
import org.apache.spark.sql.SaveMode._ | |
val sc = spark.sparkContext | |
val ssc = new StreamingContext(sc, Seconds(1)) | |
val inputPath = "/tmp/inputDir/" |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
COLLECTION=${2:-zylk} | |
SERVER=${3:-localhost} | |
PORT=${4:-8983} | |
if [ -z "$1" ]; then | |
# Usage | |
echo 'Usage: add-by-id.sh <id> [<collection> <solr-server=localhost> <port=8383>]' | |
else | |
curl -X POST "http://${SERVER}:${PORT}/solr/${COLLECTION}/update?commit=true" -H "Content-Type: text/xml" --data-binary "<add><doc><field name='id'>$1</field><field name='url'>$1</field></doc></add>" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// ==UserScript== | |
// @name Leboncoin RSS | |
// @namespace http://gist.github.com/fb7b790fb6548bdec3ec5259bebd20c0 | |
// @author Tegomass | |
// @description A kind of RSS for LeBonCoin with your personnal search | |
// @include * | |
// @require https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min.js | |
// @version 1.1 | |
// @grant GM_addStyle | |
// @grant GM_setValue |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* Generate Case class from DataFrame.schema | |
* | |
* val df:DataFrame = ... | |
* | |
* val s2cc = new Schema2CaseClass | |
* import s2cc.implicit._ | |
* | |
* println(s2cc.schemaToCaseClass(df.schema, "MyClass")) | |
* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import java.io.InputStream | |
import org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils | |
import org.apache.spark.sql.{ DataFrame, Row } | |
import org.postgresql.copy.CopyManager | |
import org.postgresql.core.BaseConnection | |
val jdbcUrl = s"jdbc:postgresql://..." // db credentials elided | |
val connectionProperties = { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WHITESPACE ODDITY | |
by Paul Phillips, in eternal admiration of David Bowie, RIP | |
Bound Ctrl to Major mode | |
Bound Ctrl to Major mode | |
Read inputrc and set extdebug on | |
Bound Ctrl to Major mode (Ten, Nine, Eight, Seven, Six) | |
Connecting readline, options on (Five, Four, Three) | |
Check the syntax, may terminfo be with you (Two, One, Exec) |
This document describes sample process of implementing part of existing Dim_Instance
ETL.
I took only Clound Block Storage source to simplify and speedup the process. I also ignnored creation of extended tables (specific for this particular ETL process). Below are code and final thoughts about possible Spark
usage as primary ETL tool.
Basic ETL implementation is really straightforward. The only real problem (I mean, really problem) is to find correct and comprehensive Mapping document (description what source fields go where).
NewerOlder