Skip to content

Instantly share code, notes, and snippets.

@matanper
matanper / DeduplicationKeySession.java
Created December 21, 2023 13:26
Apache Flink deduplication by session key
package org.monocle;
import java.io.IOException;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
@matanper
matanper / parquet_to_tsdb.py
Last active December 14, 2023 19:12
Loading parquet files from S3 into timescaledb
import boto3
import os
import time
from subprocess import Popen, PIPE
import pyarrow.parquet as pq
import pickle
import hashlib
TSDB_CPUS = 8
@matanper
matanper / main.py
Created July 27, 2023 10:56
Python blocking sync function to async function in thread decorator
def to_asyncio_in_thread(func):
async def wrapper(*args, **kwargs):
return await asyncio.get_event_loop().run_in_executor(None, lambda: func(*args, **kwargs))
return wrapper
# Usage:
@to_asyncio_in_thread
@matanper
matanper / cache.py
Created May 3, 2023 08:54
A thread-safe TTL cache function in python, and a thread-safe lock by parameter (hash key)
from cachetools import TTLCache
from cachetools.keys import hashkey
from parameterized_lock import parameterized_lock
def async_threadsafe_ttl_cache(func=None, ttl=60):
cache = TTLCache(maxsize=100, ttl=ttl)
def decorator(decorated_func):
@matanper
matanper / python_run_time_profiler.py
Created May 15, 2022 07:56
Python code snippet to run run-time profiling for python code
import cProfile
import io
import pstats
pr = cProfile.Profile()
pr.enable()
### Code to profile here
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('time')
@matanper
matanper / Dockerfile-debian
Created April 16, 2019 16:54
Debian Dockerfile
FROM debian
# Adding wanted packages
RUN apt-get update && \
apt-get install -y --fix-missing \
tcpdump \
bash-completion \
dos2unix \
telnet && \
rm -rf /var/lib/apt/lists/* && \
@matanper
matanper / Postgres-Jsquery-Dockerfile
Last active April 8, 2019 22:03
Postgres with jsquery extensions dockerfile
FROM mdillon/postgis:11-alpine
ENV JSQUERY_VERSION 1.1.1
RUN apk add --no-cache --virtual .fetch-deps \
ca-certificates \
openssl \
tar \
&& apk add --no-cache --virtual .build-deps \
g++ \
@matanper
matanper / Dockerfile-alpine
Last active April 15, 2019 14:41
Alpine with extra libs
# Using alpine version with glibc installed to run apps like Java
FROM frolvlad/alpine-glibc:alpine-3.9
# Adding wanted packages
RUN apk add --no-cache \
bash \
tcpdump \
bash-completion \
dos2unix \
busybox-extras # For telnet
@matanper
matanper / centos_openjdk11_install
Created March 26, 2019 19:59
Script to install openjdk 11.0.02 on centos/rhel
#!/bin/bash
tar zxvf openjdk-11.0.2_linux-x64_bin.tar.gz
mv jdk-11.0.2 /usr/local
echo "export JAVA_HOME=/usr/local/jdk-11.0.2" >> /etc/profile.d/jdk11.sh
echo "export PATH=$PATH:$JAVA_HOME/bin" >> /etc/profile.d/jdk11.sh
source /etc/profile.d/jdk11.sh
java -version
@matanper
matanper / debian-package-downloader
Last active March 23, 2019 18:48
Script to download packages in debian linux with all dependencies for installation in offline servers
#!/bin/bash
function contains() {
local n=$#
local value=${!n}
for ((i=1;i < $#;i++)) {
if [ "${!i}" == "${value}" ]; then
echo "y"
return 0
fi