shawel

#!/bin/bash -e

# Custom Variables
NODE_ARCH=$([ "$CI_RUNNER_EXECUTABLE_ARCH" == "linux/arm64" ] && echo "arm64v8" || echo "amd64")
echo "NODE_ARCH: $NODE_ARCH"

if [[ -z "$CI_COMMIT_TAG" ]]; then
    export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
    export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
else

shawel

# you will need to alias the internal kube IPs to your loopback interface

# get pods with IPs
kubectl get pods -n kafka -o wide

# add aliases
sudo ifconfig lo0 100.101.168.109 alias
sudo ifconfig lo0 100.117.38.251 alias
sudo ifconfig lo0 100.121.158.154 alias

shawel

package bench

import (
	"strconv"
	"testing"
)

var smallStr = "35"
var bigStr = "999999999999999"

shawel

Better local require() paths for Node.js

Problem

When the directory structure of your Node.js application (not library!) has some depth, you end up with a lot of annoying relative paths in your require calls like:

const Article = require('../../../../app/models/article');

Those suck for maintenance and they're ugly.

Possible solutions

shawel

Setup Environmnet variables for Hadoop.

export HADOOP_VERSION=2.8.5
export HADOOP_HOME=${HOME}/hadoop-$HADOOP_VERSION
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=${HADOOP_HOME}/bin:$PATH

Download Hadoop files.

shawel

/**
  * Generate Case class from DataFrame.schema
  *
  *  val df:DataFrame = ...
  *
  *  val s2cc = new Schema2CaseClass
  *  import s2cc.implicit._
  *
  *  println(s2cc.schemaToCaseClass(df.schema, "MyClass"))
  *

shawel

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...

shawel

Kafka 0.11.0.0 (Confluent 3.3.0) added support to manipulate offsets for a consumer group via cli kafka-consumer-groups command.

1. List the topics to which the group is subscribed

kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --describe

Note the values under "CURRENT-OFFSET" and "LOG-END-OFFSET". "CURRENT-OFFSET" is the offset where this consumer group is currently at in each of the partitions.

2. Reset the consumer offset for a topic (preview)

shawel

Z-Order curves are used to encode multiple dimensions to one dimension while maintaining locality. This feature makes them useful for indexing multidimensional data such as geospatial data. In BigTable-like systems (Accumulo, HBase, Cassandra a z-order curve index can translate a bounding box query to a single range scan. As this example shows, sometimes the locality properties of the curve are very good and few points outside the bounding box are scanned. Other times though, many points outside the bounding box are scanned if using a single range.

This example was inspired by Mike Bostock's Quadtree example

shawel

import {  take, put, call, fork, cancel, cancelled } from 'redux-saga/effects'

import {EVENT_POLLING_START, EVENT_POLLING_STOP} from "constants/eventPolls";
import {ClientEventType} from "constants/serverEvents";



function* handleEventA(serverEvent) {
    yield put({type : serverEvent.type, payload : {name : serverEvent.eventAData}});
}