davideicardi

Cloudera quick start with docker

This installation of Cloudera is for development only, do not use it in production.

Getting the image

Download cloudera docker image from https://www.cloudera.com/downloads/quickstart_vms/5-13.html

See also. https://www.cloudera.com/documentation/enterprise/5-13-x/topics/quickstart_docker_container.html

davideicardi

<?xml version="1.0" encoding="UTF-8" ?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

davideicardi

Scala/sbt minimal multi projects setup

Assume that you have the following directory structure:

• your-app
  • project1
    • src
      • main
        • scala
          • App.scala

davideicardi

Alpakka Kafka connector (akka-stream-kafka) example

Simple solution to use Alpakka Kafka connector to produce and consume kafka messages.

I assume that you have 2 scala apps, a producer and a consumer.

Producer

Add the following dependencies:

davideicardi

Alpakka HBase connector

In this example we create an Akka Sink that write message to HBase.

Official documentation: https://developer.lightbend.com/docs/alpakka/current/hbase.html

Add the following dependencies:

• "com.lightbend.akka" %% "akka-stream-alpakka-hbase" % "0.19"

davideicardi

/*
https://scalafiddle.io/sf/sniohcZ/3
(old version: https://scalafiddle.io/sf/sniohcZ/1)

Use PassThroughFlow when you have a message that should be used in a 
flow that trasform it but you want to maintain the original message for
another following flow.
For example if you consume messages from Kafka (CommittableMessage).
You process the message (transform, save it inside a database, ...) and then you need again the original message 
to commit the offset.

davideicardi

import akka._
import akka.stream._
import akka.stream.scaladsl._
import akka.actor._

// WARNING: Ordering is not guaranteed.
// The problem with this Graph is that when splitting the flow 
//  we obtain 2 concurrent flows. 
// The result of conditionalFlow can returns after another messages is processed in the brodcast.
// ie. If you have 10 messages from 0 to 9. Let's say that we run a conditional flow only for element 0.

davideicardi

Gradle multi-projects scala build

Assume that you have the following directory structure:

• your-app
  • project1
    • src
      • main
        • scala
• App.scala

davideicardi

Deploy VerneMq inside Kubernetes

Allow the default service account to read pods data, by adding a role binding:

TODO: Now the yaml works only for the dev1 namespace, how can I make it more generic?

kubectl apply -f ./add-read-pods-binding.yaml

davideicardi

Avro serialization

There are 4 possible serialization format when using avro:

• Avro Json encoding
• Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
• Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
• Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)