mrsimpson

Apache Flink

Apache Flink ist eine Open-Source-Stream-Processing-Engine, die für verteilte, hochparallele und fehlertolerante Datenströme entwickelt wurde. Es ermöglicht die Verarbeitung von kontinuierlichen Datenströmen in Echtzeit.

Integration mit Apache Kafka

Apache Flink integriert sich nahtlos mit Apache Kafka, dem de-facto-Standard für Datenströme:

• Flink kann Kafka-Topics als Datenquellen nutzen und Ergebnisse in Kafka-Topics schreiben[1][4]
• Flink unterstützt die exakt-einmalige Verarbeitung von Ereignissen, selbst bei Systemausfällen[4]

mrsimpson

package de.fermata.flink.app;

import java.time.Instant;
import java.util.Random;
import java.util.stream.StreamSupport;
import org.apache.flink.api.common.state.BroadcastState;
import org.apache.flink.api.common.state.MapStateDescriptor;
import org.apache.flink.api.common.state.ReadOnlyBroadcastState;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;

mrsimpson

package de.fermata.flink.app;

import java.time.Duration;
import java.util.stream.StreamSupport;
import org.apache.flink.api.common.eventtime.*;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.PrintSinkFunction;
import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;

mrsimpson

import java.sql.Timestamp;
import java.util.Comparator;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.Configuration;

mrsimpson

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
import org.apache.flink.streaming.api.watermark.Watermark;

/**

mrsimpson

Yet another Kafka compose

This Gist bundles a Kafka with useful tools and a UI to maintain it.

CAUTION Kafka is very picky about non-cleaned-up state. Thus, when stopping it, remove the volumes

docker compose down --volumes

mrsimpson

mrsimpson

function generatePassword() {
    const groups = ["ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",
        "0123456789",
        "!§$%&/()=?*+-"];

    return [12, 2, 1]
        .map(function (len, i) {
            return Array(len)
                .fill(groups[i]).map(function (x) {
                    return x[Math.floor(Math.random() * x.length)]

mrsimpson

name: CI/CD

on:
  push:
    paths-ignore:
      - '**.md'

jobs:
  test:
    runs-on: ubuntu-latest

mrsimpson

The full story

kubernetes-cluster on Raspberry PI

https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/

ssh-copy-id -i ~/.ssh/id_rsa.pub pi@<IP> -f