davideicardi

# https://github.com/wurstmeister/kafka-docker
# https://medium.com/big-data-engineering/hello-kafka-world-the-complete-guide-to-kafka-with-docker-and-python-f788e2588cfc

version: '3'
services:

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
     - "2181:2181"

davideicardi

/*
  Calculate project's version using GIT tags and commits.
  
  Main idea is to avoid having a version written in the source code but instead link it to the GIT repository.
  It is similar to sbt-git plugin but increment the PATCH part when there are additional commits.

  The version is calculated using the last tag and the last commit (using `git describe` output).
  If there is at least one commit after the last tag the PATCH part is incremented by 1
  and `-SNAPSHOT` is added as a suffix.
  Otherwise the tag is used as is.

davideicardi

#!/bin/bash
set -e

NEW_SOURCE=1
DEFAULT_SOURCE=`pacmd list-sources | grep '* index' | cut -f5 -d' '`
if [ $DEFAULT_SOURCE -eq 1 ]
then
 NEW_SOURCE=0
fi
 

davideicardi

Package a Python Lambda for AWS

Python virtual env

Create Python virtual env

python3 -m venv v-env

Activate

davideicardi

aws kinesis get-shard-iterator --stream-name YOUR_STREAM_NAME --shard-id 0 --shard-iterator-type TRIM_HORIZON

aws kinesis get-records --shard-iterator YOUR_SHARD_ITERATOR

davideicardi

const fs = require("fs");
const { Transform, PassThrough } = require('stream');

function concatStreams(streams) {
  let pass = new PassThrough();
  let waiting = streams.length;
  for (let stream of streams) {
      pass = stream.pipe(pass, {end: false});
      stream.once('end', () => --waiting === 0 && pass.emit('end'));
  }

davideicardi

Scala AES-CTR encryption Bouncy Castle example

Add the following library:

libraryDependencies += "org.bouncycastle" % "bcprov-jdk16" % "1.46"

AESCTR.scala

davideicardi

Avro serialization

There are 4 possible serialization format when using avro:

• Avro Json encoding
• Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
• Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
• Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

davideicardi

Deploy VerneMq inside Kubernetes

Allow the default service account to read pods data, by adding a role binding:

TODO: Now the yaml works only for the dev1 namespace, how can I make it more generic?

kubectl apply -f ./add-read-pods-binding.yaml

davideicardi

Gradle multi-projects scala build

Assume that you have the following directory structure:

• your-app
  • project1
    • src
      • main
        • scala
• App.scala