gaborgsomogyi

hadoop fs -Dfs.s3a.access.key=ACCESS_KEY -Dfs.s3a.secret.key=SECRET_KEY -ls s3a://bucket-name//some.log

gaborgsomogyi

import java.util.Map;
import scala.Tuple2;
import java.util.Iterator;

  Map<String, String> env = System.getenv();
  for (String envName : env.keySet()) {
    System.out.format("ENV: %s=%s%n", envName, env.get(envName));
  }
  for (String s: args) {
    System.out.println("ARGS: " + s);

gaborgsomogyi

class MockReceiver(ratePerSec: Int) extends Receiver[String](StorageLevel.MEMORY_ONLY) {

  def onStart() {
    new Thread("MockSource") {
      override def run() { receive() }
    }.start()
  }

  def onStop() {
   // There is nothing much to do as the thread calling receive()

gaborgsomogyi

# Unsecure cluster broker port: 9092
# Secure cluster broker port: 9093

# List the consumer groups
kafka-consumer-groups --bootstrap-server <broker-node>:<broker-port> --list

# From the list choose the related group
kafka-consumer-groups --bootstrap-server <broker-node>:<broker-port> --describe --group <group>

gaborgsomogyi

Kafka 0.10.2 clients are the first version that supports older brokers (0.10.0 and higher).
Before 0.10.2, clients supported newer brokers (e.g. 0.8.2 clients support 0.10.2 brokers), but not older brokers.

gaborgsomogyi

The major version bump (1.x -> 2.x) is simply because support for Java 7 has been dropped.

gaborgsomogyi

ClassLoader cl = Thread.currentThread().getContextClassLoader();
URL loc = cl.getResource("log4j.properties");
System.out.println("Actually used log4j.properties: " + loc);

gaborgsomogyi

serviceName="kafka"

This will be used to connect to other brokers for replica management and also controller requests.
This should be set to whatever principal that kafka brokers are running.

gaborgsomogyi

0.10.2.0 is the first release that promises forward compatibility from client to a newer broker.

gaborgsomogyi

log4j.logger.org.apache.kafka.common.security=TRACE