Skip to content

Instantly share code, notes, and snippets.

View sumitsu's full-sized avatar

Branden Smith sumitsu

5 followers · 9 following
View GitHub Profile
@sumitsu
sumitsu / SampleSparkAppS3Spec.scala
Last active December 14, 2019 23:29
Setting AWS S3A properties on Hadoop Configuration associated with SparkSession for local unit testing with mock S3 server
import org.apache.hadoop.fs.s3a.S3AFileSystem
private val hadoopConf = spark.sparkContext.hadoopConfiguration
hadoopConf.set("fs.s3.impl", classOf[S3AFileSystem].getName)
hadoopConf.set("fs.s3a.endpoint", "http://localhost:9999")
hadoopConf.set("fs.s3a.access.key", "abc")
hadoopConf.set("fs.s3a.secret.key", "xyz")
hadoopConf.set("fs.s3a.attempts.maximum", "3")
hadoopConf.set("fs.s3a.path.style.access", "true")
@sumitsu
sumitsu / SparkSpec.scala
Created December 14, 2019 23:13
Creating a SparkSession for unit testing
val spark: SparkSession = {
val sparkConf = new SparkConf()
.set("spark.driver.host", "127.0.0.1")
.setMaster("local[2]")
.setAppName("TestSparkApp")
val sparkSession = SparkSession.builder.config(sparkConf).getOrCreate
sparkSession.sql("set spark.sql.caseSensitive=true")
sparkSession
}
@sumitsu
sumitsu / AmazonS3TestUtil.scala
Created December 14, 2019 02:11
Creating a local-mock AmazonS3Client via AmazonS3ClientBuilder
val MockS3ServerPortEnvVar: String = "MOCK_SERVER_PORT"
val MockS3ServerPortDefault: Int = 9999
val AwsEndpointUriStr: String =
s"http://localhost:${Option(System.getenv(MockS3ServerPortEnvVar)).getOrElse(MockS3ServerPortDefault)}/"
val TestBucketName: String =
s"s3-mocktest-demo-${UUID.randomUUID.toString}"
val MockAWSAccessKey: String = "abc"
val MockAWSSecretKey: String = "zyx"
def buildLocalMockTestS3Client(): AmazonS3 = {
@sahilsk
sahilsk / kafka-cheat-sheet.md
Last active September 16, 2025 17:55 — forked from filipefigcorreia/kafka-cheat-sheet.md
Apache Kafka Cheat Sheet

Kafka Cheat Sheet

Display Topic Information

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic beacon
Topic:beacon	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: beacon	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: beacon	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
@tvlooy
tvlooy / unit.sh
Last active August 20, 2025 15:42
Bash test: get the directory of a script
#!/bin/bash
function test {
MESSAGE=$1
RECEIVED=$2
EXPECTED=$3
if [ "$RECEIVED" = "$EXPECTED" ]; then
echo -e "\033[32m✔︎ Tested $MESSAGE"
else
@christopher-hopper
christopher-hopper / vm-resize-hard-disk.md
Last active July 24, 2025 03:22
Resize a Hard Disk for a Virtual Machine provisioned using Vagrant from a Linux base box to run using VirutalBox.

Resize a Hard Disk for a Virtual Machine

Our Virtual Machines are provisioned using Vagrant from a Linux base box to run using VirutalBox. If the Hard Disk space runs out and you cannot remove files to free-up space, you can resize the Hard Disk using some VirtualBox and Linux commands.

Some assumptions

The following steps assume you've got a set-up like mine, where:

@lelandbatey
lelandbatey / whiteboardCleaner.md
Last active May 20, 2025 13:11
Whiteboard Picture Cleaner - Shell one-liner/script to clean up and beautify photos of whiteboards!

Description

This simple script will take a picture of a whiteboard and use parts of the ImageMagick library with sane defaults to clean it up tremendously.

The script is here:

#!/bin/bash
convert "$1" -morphology Convolve DoG:15,100,0 -negate -normalize -blur 0x1 -channel RBG -level 60%,91%,0.1 "$2"

Results