erikdw

// Generated by the protocol buffer compiler.  DO NOT EDIT!
// source: include/mesos/mesos.proto

package org.apache.mesos;

public final class Protos {
  private Protos() {}
  public static void registerAllExtensions(
      com.google.protobuf.ExtensionRegistry registry) {
  }

erikdw

Tuning Storm+Trident

Tuning a dataflow system is easy:

The First Rule of Dataflow Tuning:
* Ensure each stage is always ready to accept records, and
* Deliver each processed record promptly to its destination

erikdw

#!/bin/bash
set -u
set -e -o pipefail

version=${1:-}
# `g++ --version` should reveal that GCC is 4.8.2
gcc_version=`g++ --version`
echo "Building mesos version ${version} using g++ version ${gcc_version}"

# now download and build mesos

erikdw

Adding 2 nodes to an existing 3-node ZooKeeper ensemble without losing the Quorum

Since many deployments may start out with 3 nodes and so little is known about how to grow a cluster from 3 memebrs to 5 members without losing the existing Quorum, here is an example of how this might be achieved.

In this example, all 5 nodes will be running on the same Vagrant host for the purpose of illustration, running on distinct configurations (ports and data directories) without the actual load of clients.

YMMV. Caveat usufructuarius.

Step 1: Have a healthy 3-node ensemble

erikdw

2016-06-19T07:55:05.924+0000 s.m.u.MesosCommon [INFO] Available resources at workerhostname: 
  cpu: Resource cpus - Total available : 19.500000 Total available by reservation type : [ , [DYNAMICALLY_RESERVED : 0.0, STATICALLY_RESERVED : 0.0, UNRESERVED : 19.5] ], 
  memory: Resource mem - Total available : 88561.000000 Total available by reservation type : [ , [DYNAMICALLY_RESERVED : 0.0, STATICALLY_RESERVED : 0.0, UNRESERVED : 88561.0] ],
  ports : [31003-31003,31005-32000]

Example of new log:

2016-06-19T09:21:43.075+0000 s.m.u.MesosCommon [INFO] Available resources at workerhostname: 

erikdw

import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;

public class HelloCovariance {
  public static void main(String[] args) {
    ConcurrentHashMap<String, String> properties = new ConcurrentHashMap<>();
    Set<String> keySet = properties.keySet();
  }
}

erikdw

diff --git Vagrantfile Vagrantfile
index bbbca8b..775fada 100755
--- Vagrantfile
+++ Vagrantfile
@@ -65,7 +65,7 @@ fi
 install_package "libcurl3"
 install_package "zookeeperd"
 install_package "aria2"
-install_package "mesos=0.25.0-0.2.70.ubuntu1404"
+install_package "mesos=1.0.1-2.0.93.ubuntu1404"

erikdw

% cd storm
% mvn validate
% (find . -name 'checkstyle-result.xml' -exec cat {} \;) | grep 'consecutive capital letters' | cut -d';' -f2 | cut -d'&' -f1 | sort | uniq -c | sort

erikdw

#!/bin/bash
IMAGE="mjanser/capifony"
CURRDIR=$(pwd)

if [ "$(uname)" == "Darwin" ]; then

    # under Mac OS X platform
    command -v jq >/dev/null 2>&1 || { echo >&2 "please install jq with \"brew install jq\" or \"port install jq\""; exit 1; }
    docker pull $IMAGE
    ENTRYPOINT=$(docker inspect $IMAGE | jq -r '.[0].ContainerConfig.Entrypoint[0]')

erikdw

Metaspace Research

What is PermGen?

• The Permanent Generation memory pool contains permanent class metadata and descriptors information when classes are loaded
  • PermGen space is always reserved for classes and items that are attached to them (i.e., static members)
• PermGem space is contiguous in memory to the Java heap and have the same rounds of Garbage Collection but it is not part of the Java heap

What is Metaspace?

• In JDK8, the PermGen space has been entirely replaced by Metaspace which is no longer contiguous to the Java heap and now exists in native memory