gaborgsomogyi

sudo kill -9 `ps ax | grep 'coreaudio[a-z]' | awk '{print $1}'`

gaborgsomogyi

[gaborsomogyi:~/certs] 34s 130 $ keytool -genkey -keyalg RSA -alias spark -keystore spark-keystore.jks -validity 3650 -keysize 2048
Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  gsomogyi-MBP16
What is the name of your organizational unit?
  [Unknown]:  Engineering
What is the name of your organization?
  [Unknown]:  Cloudera
What is the name of your City or Locality?

gaborgsomogyi

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version

Release: Broker protocol - 0.10.0, Java clients - 0.10.2

gaborgsomogyi

Configure kerberos on all nodes:
$ cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 renew_lifetime = 90m
 default_realm = DT1GCE.CLOUDERA.COM

gaborgsomogyi

kafka-topics --zookeeper localhost:2181 --delete --topic topic1

gaborgsomogyi

$ cat consumer.properties
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/etc/cdep-ssl-conf/CA_STANDARD/truststore.jks
ssl.truststore.password=cloudera

$ cat jaas.conf
KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true

gaborgsomogyi

Parameters are arriving to source/sink lowercase.

maxOffsetsPerTrigger parameter as an example:

• KafkaSourceProvider uses caseInsensitiveParams which converts keys to lowercase
• KafkaMicroBatchStream uses CaseInsensitiveStringMap where get operation uses lowercase conversion
• KafkaSource uses CaseInsensitiveMap where get operation uses lowercase conversion

In the last case CaseInsensitiveMap extends Map and as said it provides lowercase key lookup but in the [interface](https://github.com/apache/spark/blob/3e4

gaborgsomogyi

• The old scala API: kafka.(consumer|producer)
• The new java API: org.apache.kafka.(consumer|producer)

gaborgsomogyi

$ spark-shell
> spark.sql("SET -v").show(999, false)

gaborgsomogyi

rm -rf .git/gitk.cache