Skip to content

Instantly share code, notes, and snippets.

View zorteran's full-sized avatar

Maciej Szymczyk zorteran

View GitHub Profile
@zorteran
zorteran / pom.xml
Created May 12, 2020 05:05
part of pom.xml - package fat jar
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
@zorteran
zorteran / EnvTools.java
Last active May 12, 2020 05:13
EnvTools
public class EnvTools {
public static final String INPUT_TOPIC = "INPUT_TOPIC";
public static final String OUTPUT_TOPIC = "OUTPUT_TOPIC";
public static final String APPLICATION_ID_CONFIG = "APPLICATION_ID_CONFIG";
public static final String BOOTSTRAP_SERVERS_CONFIG = "BOOTSTRAP_SERVERS_CONFIG";
public static String getEnvValue(String environmentKey, String defaultValue)
{
String envValue = System.getenv(environmentKey);
if(envValue != null &amp;&amp; !envValue.isEmpty())
@zorteran
zorteran / ZtmStream.java
Created May 12, 2020 05:15
Part of ZtmStream.java
...
public ZtmStream() {
inputTopic = EnvTools.getEnvValue(EnvTools.INPUT_TOPIC, "ztm-input");
outputTopic = EnvTools.getEnvValue(EnvTools.OUTPUT_TOPIC, "ztm-output");
}
...
private static Properties createProperties() {
Properties props = new Properties();
String appIdConfig = EnvTools.getEnvValue(EnvTools.APPLICATION_ID_CONFIG, "wiaderko-ztm-stream");
String bootstrapServersConfig = EnvTools.getEnvValue(EnvTools.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
@zorteran
zorteran / Dockerfile
Created May 12, 2020 05:20
Kafka Streams Dockerfile
FROM maven:3-openjdk-8-slim AS BUILD
COPY pom.xml /tmp/
COPY src /tmp/src/
WORKDIR /tmp/
RUN mvn package
FROM openjdk:8-alpine AS RUNTIME
RUN apk update && apk add --no-cache libc6-compat
RUN ln -s /lib64/ld-linux-x86-64.so.2 /lib/ld-linux-x86-64.so.2
WORKDIR /app/
@zorteran
zorteran / docker-compose.yml
Created May 12, 2020 05:30
Kafka + Zookeeper + ZtmStream Kafka Stream
version: '3.3'
services:
zookeeper:
image: 'bitnami/zookeeper:3'
ports:
- '2181:2181'
volumes:
- 'zookeeper_data:/bitnami'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
@zorteran
zorteran / ztm_to_kafka.py
Created June 1, 2020 04:44
ZTM Api to Kafka script
import requests
import json
import time
from kafka import KafkaProducer
token = 'use your own API Token'
url = 'https://api.um.warszawa.pl/api/action/busestrams_get/'
resource_id = 'f2e5503e927d-4ad3-9500-4ab9e55deb59'
sleep_time = 15
@zorteran
zorteran / keepalived.conf
Created June 30, 2020 11:33
keepalived simple config
vrrp_instance VI_1 {
interface ens33
state MASTER
virtual_router_id 51
priority 101
virtual_ipaddress {
192.168.10.166
}
}
@zorteran
zorteran / logstash_check.sh
Last active August 29, 2020 06:27
Checking if logstash is alive
#!/bin/bash
logstash_response="$(curl --silent localhost:9600)"
if echo $logstash_response | grep -q '"status":"green"'
then
echo "Logstash is green. It's ok :-)"
exit 0
else
echo "Logstash is not green :-( plz help"
exit 1
fi
@zorteran
zorteran / keepalived.conf
Created July 6, 2020 04:38
Keepalived script + logstash check
global_defs {
enable_script_security
}
vrrp_script chk_logstash {
script "/etc/keepalived/lol.sh" # path of the script to execute
interval 1 # seconds between script invocations
timeout 1 # seconds after which script is considered to have failed
@zorteran
zorteran / simple_etl_to_cassandra.ipynb
Created September 5, 2020 18:00
Simple ETL joining data from MySQL and MongoDB and wriiting to Cassandra
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.