gaborgsomogyi

sudo kill -9 `ps ax | grep 'coreaudio[a-z]' | awk '{print $1}'`

gaborgsomogyi

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version

Release: Broker protocol - 0.10.0, Java clients - 0.10.2

gaborgsomogyi

kafka-topics --zookeeper localhost:2181 --delete --topic topic1

gaborgsomogyi

$ cat consumer.properties
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/etc/cdep-ssl-conf/CA_STANDARD/truststore.jks
ssl.truststore.password=cloudera

$ cat jaas.conf
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true

gaborgsomogyi

Parameters are arriving to source/sink lowercase.

maxOffsetsPerTrigger parameter as an example:

• KafkaSourceProvider uses caseInsensitiveParams which converts keys to lowercase
• KafkaMicroBatchStream uses CaseInsensitiveStringMap where get operation uses lowercase conversion
• KafkaSource uses CaseInsensitiveMap where get operation uses lowercase conversion

In the last case CaseInsensitiveMap extends Map and as said it provides lowercase key lookup but in the [interface](https://github.com/apache/spark/blob/3e4

gaborgsomogyi

• The old scala API: kafka.(consumer|producer)
• The new java API: org.apache.kafka.(consumer|producer)

gaborgsomogyi

$ spark-shell
> spark.sql("SET -v").show(999, false)

gaborgsomogyi

rm -rf .git/gitk.cache

gaborgsomogyi

ContainerId string format is changed if RM restarts with work-preserving recovery enabled.

It used to be such format:
container_{clusterTimestamp}_{appId}_{attemptId}_{containerId}
e.g.: container_1410901177871_0001_01_000005.

It is now changed to:
container_e{epoch}_{clusterTimestamp}_{appId}_{attemptId}_{containerId}
e.g.: container_e17_1410901177871_0001_01_000005.

gaborgsomogyi

Lag calculation

The lag in Kafka topic calculated in the following way: endOffsets - committedOffsets

endOffsets

To fetch this the following steps can be made:

• Get available TopicPartitions with AdminClient describeTopics(java.util.Collection<java.lang.String> topicNames) API.
• Create a consumer and fetch the endOffsets with endOffsets(java.util.Collection<TopicPartition> partitions) API.