davideicardi

Cloudera quick start with docker

This installation of Cloudera is for development only, do not use it in production.

Getting the image

Download cloudera docker image from https://www.cloudera.com/downloads/quickstart_vms/5-13.html

See also. https://www.cloudera.com/documentation/enterprise/5-13-x/topics/quickstart_docker_container.html

davideicardi

# https://github.com/wurstmeister/kafka-docker
# https://medium.com/big-data-engineering/hello-kafka-world-the-complete-guide-to-kafka-with-docker-and-python-f788e2588cfc

version: '3'
services:

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
     - "2181:2181"

davideicardi

	/// <summary>
	/// A Timer that execute an async callback after a due time and repeating it after a period (interval).
	/// Callback is a non reentrant timer (ie. callbacks never overlaps). Dispose method wait for the current callback to finish.
	/// Timer can be cancelled using a CancellationToken or by calling StopAsync or by calling Dispose.
	/// Exception inside callbacks are ignored and just traced.
	/// Callback is invoked at each interval (period) after the end of the previous invocation.
	/// </summary>
	public sealed class TimerAsync : IDisposable
	{
		private readonly Func<CancellationToken, Task> _callback;

davideicardi

import { Bucket } from 'aws-cdk-lib/aws-s3';
import { BucketDeployment, Source } from 'aws-cdk-lib/aws-s3-deployment';
import { CloudFrontWebDistribution, OriginProtocolPolicy } from 'aws-cdk-lib/aws-cloudfront';
import { ARecord, HostedZone, RecordTarget } from 'aws-cdk-lib/aws-route53';
import { CloudFrontTarget } from 'aws-cdk-lib/aws-route53-targets';
import { Construct } from 'constructs';
import { aws_s3 } from 'aws-cdk-lib';

interface StaticWebsiteProps {
  domainName: string;

davideicardi

Publish

Add the following configuration to your build.sbt:

  // publish to github packages settings
  publishTo := Some("GitHub <GITHUB_OWNER> Apache Maven Packages" at "https://maven.pkg.github.com/<GITHUB_OWNER>/<GITHUB_PROJECT>"),
  publishMavenStyle := true,
  credentials += Credentials(
    "GitHub Package Registry",

davideicardi

// Javascript helper functions for parsing and displaying UUIDs in the MongoDB shell.
// This is a temporary solution until SERVER-3153 is implemented.
// To create BinData values corresponding to the various driver encodings use:
//      var s = "{00112233-4455-6677-8899-aabbccddeeff}";
//      var uuid = UUID(s); // new Standard encoding
//      var juuid = JUUID(s); // JavaLegacy encoding
//      var csuuid = CSUUID(s); // CSharpLegacy encoding
//      var pyuuid = PYUUID(s); // PythonLegacy encoding
// To convert the various BinData values back to human readable UUIDs use:
//      uuid.toUUID()     => 'UUID("00112233-4455-6677-8899-aabbccddeeff")'

davideicardi

# Create a container from the mongo image, 
#  run is as a daemon (-d), expose the port 27017 (-p),
#  set it to auto start (--restart)
#  and with mongo authentication (--auth)
# Image used is https://hub.docker.com/_/mongo/
docker pull mongo
docker run --name YOURCONTAINERNAME --restart=always -d -p 27017:27017 mongo mongod --auth

# Using the mongo "localhost exception" (https://docs.mongodb.org/v3.0/core/security-users/#localhost-exception) 
# add a root user

davideicardi

Gradle multi-projects scala build

Assume that you have the following directory structure:

• your-app
  • project1
    • src
      • main
        • scala
• App.scala

davideicardi

Avro serialization

There are 4 possible serialization format when using avro:

• Avro Json encoding
• Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
• Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
• Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

davideicardi

const fs = require("fs");
const { Transform, PassThrough } = require('stream');

function concatStreams(streams) {
  let pass = new PassThrough();
  let waiting = streams.length;
  for (let stream of streams) {
      pass = stream.pipe(pass, {end: false});
      stream.once('end', () => --waiting === 0 && pass.emit('end'));
  }