davideicardi

import akka._
import akka.stream._
import akka.stream.scaladsl._
import akka.actor._

// WARNING: Ordering is not guaranteed.
// The problem with this Graph is that when splitting the flow 
//  we obtain 2 concurrent flows. 
// The result of conditionalFlow can returns after another messages is processed in the brodcast.
// ie. If you have 10 messages from 0 to 9. Let's say that we run a conditional flow only for element 0.

davideicardi

/*
https://scalafiddle.io/sf/sniohcZ/3
(old version: https://scalafiddle.io/sf/sniohcZ/1)

Use PassThroughFlow when you have a message that should be used in a 
flow that trasform it but you want to maintain the original message for
another following flow.
For example if you consume messages from Kafka (CommittableMessage).
You process the message (transform, save it inside a database, ...) and then you need again the original message 
to commit the offset.

davideicardi

Alpakka HBase connector

In this example we create an Akka Sink that write message to HBase.

Official documentation: https://developer.lightbend.com/docs/alpakka/current/hbase.html

Add the following dependencies:

• "com.lightbend.akka" %% "akka-stream-alpakka-hbase" % "0.19"

davideicardi

Alpakka Kafka connector (akka-stream-kafka) example

Simple solution to use Alpakka Kafka connector to produce and consume kafka messages.

I assume that you have 2 scala apps, a producer and a consumer.

Producer

Add the following dependencies:

davideicardi

Scala/sbt minimal multi projects setup

Assume that you have the following directory structure:

• your-app
  • project1
    • src
      • main
        • scala
          • App.scala

davideicardi

<?xml version="1.0" encoding="UTF-8" ?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

davideicardi

Cloudera quick start with docker

This installation of Cloudera is for development only, do not use it in production.

Getting the image

Download cloudera docker image from https://www.cloudera.com/downloads/quickstart_vms/5-13.html

See also. https://www.cloudera.com/documentation/enterprise/5-13-x/topics/quickstart_docker_container.html

davideicardi

// This method convert a json to a valid string json
// It should handle all mongodb data types correctly (Date, ObjectId, ...) thanks to the
// Newtonsoft.Json.Bson.BsonReader serializer

public string ToJson(BsonDocument bson)
{
	using (var stream = new MemoryStream())
	{
		using (var writer = new BsonBinaryWriter(stream))
		{

davideicardi

Convert a .NET 4.x project to .NET Core

Steps to convert .csproj to new VS 2017 format (.NET 4.5 to .NET Core)

• delete any files inside the project folder but not included in the project
• unload project from visual studio
• edit .csproj
• a .csproj template with nuget packages, nuget info, code analysis, style cop:

davideicardi

Development mongodb cluster tutorial

Tutorial on how to create a development mongodb cluster. Based on: https://docs.mongodb.com/manual/tutorial/deploy-replica-set-for-testing/

Start mongo1:

mongod --config d:\data\mongodb\mongod-cluster1.cfg