Skip to content

Instantly share code, notes, and snippets.

View yaravind's full-sized avatar
💭
Constraints Liberate. Liberties Constrain.

Aravind Yarram yaravind

💭
Constraints Liberate. Liberties Constrain.
47 followers · 7 following
View GitHub Profile
@yaravind
yaravind / install-docker-ce-on-elementaryos-juno.sh
Last active December 28, 2018 04:32 — forked from BeerOnBeard/install-docker-ce-on-elementaryos-loki.sh
Install Docker CE on ElementaryOS 0.4.1 Loki
#!/bin/bash
set -e
##########################################################
# Install script for Docker-CE on ElementaryOS 0.4.1 Loki
# Had to update the repository to point to xenial instead
# of using 'lsb_release -cs' because there's no loki
# repository at download.docker.com.
##########################################################
@yaravind
yaravind / 00-LogParser-Hive-Regex
Created May 18, 2018 02:41 — forked from airawat/00-LogParser-Hive-Regex
Log parser in Hive using regex serde
This gist includes hive ql scripts to create an external partitioned table for Syslog
generated log files using regex serde;
Usecase: Count the number of occurances of processes that got logged, by year, month,
day and process.
Includes:
---------
Sample data and structure: 01-SampleDataAndStructure
Data download: 02-DataDownload
Data load commands: 03-DataLoadCommands
@yaravind
yaravind / SparkSQLJira.scala
Created May 31, 2017 14:49 — forked from crocker/SparkSQLJira.scala
package com.databricks.spark.jira
import scala.io.Source
import org.apache.spark.rdd.RDD
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.sources.{TableScan, BaseRelation, RelationProvider}
@yaravind
yaravind / spark-duplicates.scala
Created May 31, 2017 14:39 — forked from crocker/spark-duplicates.scala
Find duplicates in a Spark DataFrame
val transactions = spark.read
.option("header", "true")
.option("inferSchema", "true")
.json("s3n://bucket-name/transaction.json")
transactions.groupBy("id", "organization").count.sort($"count".desc).show
@yaravind
yaravind / spark-compare-dataframes.scala
Created May 31, 2017 14:38 — forked from crocker/spark-compare-dataframes.scala
val today = LocalDate.now
val todayTransactions = spark.read
.option("header", "true")
.option("inferSchema", "true")
.json(s"s3n://bucket-name/${today}/transaction.json")
val yesterdayTransactions = spark.read
.option("header", "true")
.option("inferSchema", "true")
@yaravind
yaravind / spark-rest-submit.sh
Last active October 30, 2020 02:43
Submit apps (SparkPi as e.g.) to spark cluster using rest api
curl -X POST -d http://master-host:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
"action": "CreateSubmissionRequest",
"appResource": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"clientSparkVersion": "2.0.0",
"appArgs": [ "10" ],
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
@yaravind
yaravind / db-example.go
Created January 21, 2017 16:06
Sample SQLite database app
package main
import (
"database/sql"
"fmt"
_ "github.com/mattn/go-sqlite3"
)
func main() {
@yaravind
yaravind / account_main.go
Created January 9, 2017 16:22
func main() {
aliceAcct := OpenSavingsAccount("12345", "Alice", time.Date(1999, time.January, 03, 0, 0, 0, 0, time.UTC))
fmt.Println("Alice's account =", aliceAcct)
aliceAcct.Deposit(Money(100.0))
fmt.Println("Alice's account (after deposit) =", aliceAcct)
if err := aliceAcct.Withdraw(Money(10)); err != nil {
fmt.Println(err)
} else {
@yaravind
yaravind / account.go
Last active January 9, 2017 02:38
Account management domain model
package account
import (
"errors"
"fmt"
"time"
)
//Account is an interface that wraps the common behavior for accounts.
type Account interface {
@yaravind
yaravind / Unix Shell related commands
Last active December 29, 2016 22:47
Shell commands to know details on the Shell