bernhardschaefer

#!/bin/bash

# Minimum TODOs on a per job basis:
# 1. define name, application jar path, main class, queue and log4j-yarn.properties path
# 2. remove properties not applicable to your Spark version (Spark 1.x vs. Spark 2.x)
# 3. tweak num_executors, executor_memory (+ overhead), and backpressure settings

# the two most important settings:
num_executors=6
executor_memory=3g

kaathewise

#/usr/bin/env perl -wl
use strict;

use List::Util qw(sum);

my %graph;
<>;
while (<>) {
  my ($a, $b) = split;
  push @{$graph{$a}}, $b;

timvw

val builder = new KStreamBuilder()
val twitterStream: KStream[String, String] = builder.stream[String, String]("twitter")

import be.icteam.demo.{KafkaStreamWrapper => KSW}

val pipeline =
  KSW.filter { (k: String, v: String) => v.contains("apache")} _ andThen
  KSW.filter((k, v) => true) andThen
  KSW.foreach((k, v) => println(v))

vkroz

Kafka frequent commands

Assuming that the following environment variables are set:

• KAFKA_HOME where Kafka is installed on local machine (e.g. /opt/kafka)
• ZK_HOSTS identifies running zookeeper ensemble, e.g. ZK_HOSTS=192.168.0.99:2181
• KAFKA_BROKERS identifies running Kafka brokers, e.g. KAFKA_BROKERS=192.168.0.99:9092

Server

Start Zookepper and Kafka servers

JonCole

Debugging Redis Keyspace Misses

Simply put, a keyspace "MISS" means some piece of data you tried to retrieve from Redis was not there. This usually means that one of the following things happened:

1. The key expired
2. They key was renamed
3. The key was deleted
4. The key was evicted due to memory pressure
5. The entire database was flushed
6. The key was never actually inserted

petehunt

import time

class Bucket(object):
  def __init__(self, max_amount, refill_time, refill_amount):
    self.max_amount = max_amount
    self.refill_time = refill_time
    self.refill_amount = refill_amount
    self.reset()

  def _refill_count(self):

jdegoes

Applied Functional Programming with Scala - Notes

Copyright © 2016-2018 Fantasyland Institute of Learning. All rights reserved.

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x

densh

Scoped Implicit Lifetimes

All things considered, our experience in Scala Native has shown that resource management in Scala is way harder than it should be. This gist presents a simple design pattern that makes it resource management absolutely hassle-free: scoped implicit lifetimes.

The main idea behind it is to encode resource lifetimes through a concept of an implicit scope. Scopes are necessary to acquire resources. They are responsible for disposal of the resources once the evaluation exits the

jdegoes

sealed trait Free[F[_], A] { self =>
  final def map[B](ab: A => B): Free[F, B] = Free.flatMap(self, ab andThen (Free.point[F, B](_)))
  final def flatMap[B](afb: A => Free[F, B]): Free[F, B] = Free.flatMap(self, afb)
  final def interpret[G[_]: Monad](fg: F ~> G): G[A] = self match {
    case Free.Point(a0) => a0().point[G]
    case Free.Effect(fa) => fg(fa)
    case fm : Free.FlatMap[F, A] =>
    	val ga0 = fm.fa.interpret[G](fg)
    	ga0.flatMap(a0 => fm.afb(a0).interpret[G](fg))
  }

leonardofed

A curated list of AWS resources to prepare for the AWS Certifications

A curated list of awesome AWS resources you need to prepare for the all 5 AWS Certifications. This gist will include: open source repos, blogs & blogposts, ebooks, PDF, whitepapers, video courses, free lecture, slides, sample test and many other resources.

---