Skip to content

Instantly share code, notes, and snippets.

View pcejrowski's full-sized avatar

Paweł Cejrowski pcejrowski

  • Sopot, Poland
  • 10:59 (UTC +01:00)
View GitHub Profile

Debugging locally run Spark application in IntelliJ

  • Add the following lines to your spark-submit command
--num-executors 1 \
--executor-cores 1 \
--conf "spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=n" \
--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 \
  • Configure Run/Debug Configuration in Intellij: Remote -> default settings for localhost
@pcejrowski
pcejrowski / git_ammend_old.sh
Created October 16, 2017 09:29
Git ammending old commits
#!/bin/bash
# Usage:
# ./git_ammend_old.sh <commit-sha>
# then we change pick to edit for this commit in interactive mode
set -evx
git stash && \
echo "REMEBER TO CHANGE THE COMMIT TO EDIT" && \
git rebase --interactive "${1}^" && \
@pcejrowski
pcejrowski / iterm2-solarized.md
Created November 17, 2017 21:46 — forked from kevin-smets/iterm2-solarized.md
iTerm2 + Oh My Zsh + Solarized color scheme + Meslo powerline font + [Powerlevel9k] - (macOS)

Default

Default

Powerlevel9k

Powerlevel9k

@pcejrowski
pcejrowski / EternalInMemoryCache.scala
Created January 10, 2018 13:36
Simple in-memory cache in Scala
import java.util.concurrent.ConcurrentHashMap
class EternalInMemoryCache[A, B]{
private val underlying = new ConcurrentHashMap[A,B]()
def cached(key: A)(value: => B): B = {
Option(underlying.get(key)) match {
case Some(v) => v
case None =>
underlying.put(key, value)
value
}
@pcejrowski
pcejrowski / install.R
Created February 13, 2018 14:35
Installing R packages skipping those already installed
pkgs <- c("RCurl", "jsonlite", "statmod", "devtools", "roxygen2", "testthat")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) install.packages(pkg)
}
@pcejrowski
pcejrowski / DataprocPlugin.scala
Created April 19, 2018 13:35
Dataproc plugin for sbt allowing to spin-up and tear down clusters around integration tests.
import sbt.Keys._
import sbt._
object DataprocPlugin extends AutoPlugin {
object autoImport {
val dataprocZone = settingKey[String]("Zone to run Dataproc cluster in")
val dataprocClusterName = settingKey[String]("Dataproc cluster name")
val dataprocServiceAccount = settingKey[String]("Service account to use for Dataproc cluster")
val createCluster = taskKey[Unit]("dataproc-start")
@pcejrowski
pcejrowski / ModuleEndpoints.scala
Created May 9, 2018 20:18
Play Embedded Server with routings in different submodules
import play.api.mvc._
import play.api.routing.sird._
import play.api.routing.{Router, SimpleRouter}
object ModuleEndpoints extends SimpleRouter {
def routes: Router.Routes = {
case GET(p"/getendpoint") => Action { req =>
Results.Ok("/getendpoint in ModuleEndpoints")
}
}
@pcejrowski
pcejrowski / farm-fingerprint.md
Last active March 30, 2022 20:46
Farm Fingerprint function

Farm Fingerprint in different languages

BigQuery SQL

The FARM_FINGERPRINT(value) functioncomputes the fingerprint of the STRING or BYTES input using the Fingerprint64 function from the open-source FarmHash library. The output of this function for a particular input will never change.

select FARM_FINGERPRINT("alphabet")
@pcejrowski
pcejrowski / log.sh
Last active July 31, 2022 01:14
Logging in bash
NC='\033[0m'
function log_info {
green "INFO" "${1}"
}
function log_warn {
orange "WARN$" "${1}"
}
{
"public_identifier": "pcejrowski",
"profile_pic_url": "https://s3.us-west-000.backblazeb2.com/proxycurl/person/pcejrowski/profile?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0004d7f56a0400b0000000001%2F20230815%2Fus-west-000%2Fs3%2Faws4_request&X-Amz-Date=20230815T204707Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=675c967559b2a2800ad5077401287bdbd373388dee9e87b630ccd78c7c2134f7",
"background_cover_image_url": null,
"first_name": "Pawel",
"last_name": "Cejrowski",
"full_name": "Pawel Cejrowski",
"follower_count": null,
"occupation": "Senior Software Engineering Consultant at Tapad",
"headline": "Software Engineer",