Skip to content

Instantly share code, notes, and snippets.

View gaborgsomogyi's full-sized avatar

Gabor Somogyi gaborgsomogyi

View GitHub Profile
class MockReceiver(ratePerSec: Int) extends Receiver[String](StorageLevel.MEMORY_ONLY) {
def onStart() {
new Thread("MockSource") {
override def run() { receive() }
}.start()
}
def onStop() {
// There is nothing much to do as the thread calling receive()
hadoop fs -Dfs.s3a.access.key=ACCESS_KEY -Dfs.s3a.secret.key=SECRET_KEY -ls s3a://bucket-name//some.log
import java.util.Map;
import scala.Tuple2;
import java.util.Iterator;
Map<String, String> env = System.getenv();
for (String envName : env.keySet()) {
System.out.format("ENV: %s=%s%n", envName, env.get(envName));
}
for (String s: args) {
System.out.println("ARGS: " + s);
# Unsecure cluster broker port: 9092
# Secure cluster broker port: 9093
# List the consumer groups
kafka-consumer-groups --bootstrap-server <broker-node>:<broker-port> --list
# From the list choose the related group
kafka-consumer-groups --bootstrap-server <broker-node>:<broker-port> --describe --group <group>
Kafka 0.10.2 clients are the first version that supports older brokers (0.10.0 and higher).
Before 0.10.2, clients supported newer brokers (e.g. 0.8.2 clients support 0.10.2 brokers), but not older brokers.
The major version bump (1.x -> 2.x) is simply because support for Java 7 has been dropped.
ClassLoader cl = Thread.currentThread().getContextClassLoader();
URL loc = cl.getResource("log4j.properties");
System.out.println("Actually used log4j.properties: " + loc);
serviceName="kafka"
This will be used to connect to other brokers for replica management and also controller requests.
This should be set to whatever principal that kafka brokers are running.
0.10.2.0 is the first release that promises forward compatibility from client to a newer broker.
log4j.logger.org.apache.kafka.common.security=TRACE