Skip to content

Instantly share code, notes, and snippets.

View gaborgsomogyi's full-sized avatar

Gabor Somogyi gaborgsomogyi

27 followers · 10 following
View GitHub Profile
@gaborgsomogyi
gaborgsomogyi / YARN ContainerId
Last active April 26, 2019 08:48
ContainerId string format is changed if RM restarts with work-preserving recovery enabled.
It used to be such format:
container_{clusterTimestamp}_{appId}_{attemptId}_{containerId}
e.g.: container_1410901177871_0001_01_000005.
It is now changed to:
container_e{epoch}_{clusterTimestamp}_{appId}_{attemptId}_{containerId}
e.g.: container_e17_1410901177871_0001_01_000005.
@gaborgsomogyi
gaborgsomogyi / Spark 3.0 Structured Streaming Kafka consumer groupId.md
Last active October 9, 2020 07:26

Common:

  • val groupIdPrefix = spark-kafka-sources or configured with kafka.groupIdPrefix

Driver:

  • var nextId = 0
  • s"${groupIdPrefix}-${UUID.randomUUID}-${metadataPath.hashCode}-driver-${nextId}"
  • nextId += 1

Executor:

  • s"${groupIdPrefix}-${UUID.randomUUID}-${metadataPath.hashCode}-executor"
@gaborgsomogyi
gaborgsomogyi / Kafka lag calculation.md
Last active May 20, 2019 11:06
@gaborgsomogyi
gaborgsomogyi / Number of changes in a git repo.md
Created June 26, 2019 15:48
  • Using Gawk:

git log --author="Your_Name_Here" --pretty=tformat: --numstat | gawk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s removed lines: %s total lines: %s\n", add, subs, loc }' -

  • Using Awk on Mac OSX:

git log --author="Your_Name_Here" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' -

@gaborgsomogyi
gaborgsomogyi / Spark kafka-clients related changes.md
Last active March 19, 2022 09:38
@gaborgsomogyi
gaborgsomogyi / Meld on windows
Created August 29, 2019 21:02
git config --global merge.tool meld
git config --global diff.tool meld
git config --global mergetool.meld.path "C:\Program Files (x86)\Meld\meld\meld.exe"
@gaborgsomogyi
gaborgsomogyi / Single node streaming test.md
Last active October 14, 2019 15:01 
class KafkaSinkStreamingSuite
...
  test("single node streaming") {
    val input = MemoryStream[String]
    val topic = newTopic()
    testUtils.createTopic(topic)

    val writer = createKafkaWriter(
 input.toDF(),
@gaborgsomogyi
gaborgsomogyi / Single node batch test.md
Last active October 14, 2019 13:30 
class KafkaSinkBatchSuiteV2
...
  test("single node batch") {
    val topic = newTopic()
    testUtils.createTopic(topic)
    val rand = new Random()
    val data = Seq.fill(100000)(Row(topic, rand.nextInt().toString))

 val df = spark.createDataFrame(
@gaborgsomogyi
gaborgsomogyi / Howto build spark docker integration tests.md
Last active January 30, 2020 16:01 
cd external/docker-integration-tests
mvn install -DskipTests -Dscalastyle.skip=true -Dcheckstyle.skip
  • Go to IntelliJ and recompile docker-integration-tests project.
  • Start test from IntelliJ
@gaborgsomogyi
gaborgsomogyi / Apache hadoop standalone minikdc.md
Created February 2, 2020 14:17 
# Setup minikdc
cat << 'EOF' > minikdc_deps.gradle
apply plugin: 'java'

repositories {
   mavenCentral()
}

dependencies {