export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"
Use this jaas conf file
## Consumer Throughput: Single consumer thread, no compression | |
## Consumer Throughput: 3 consumer thread, no compression | |
bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \ | |
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \ | |
--messages 15000000 \ | |
--threads 1 |
export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"
Use this jaas conf file
KafkaConnect
Project Dirs | |
bin : Scripts to manage kafka and Zookeeper | |
checkstyle : Checkstyle configurations | |
clients : Kafka client code [Producer,Consumer, Admin tools classes] | |
config : Default Config files | |
connect : Kafka Connect module | |
core : Kafka server code | |
docs : Documentations |
1) PACKAGE RUNTIME DEPENDENCIES IN LIB FOLDER INSIDE PACKAGED JAR | |
<build> | |
<plugins> | |
<plugin> | |
<groupId>org.apache.maven.plugins</groupId> | |
<artifactId>maven-dependency-plugin</artifactId> | |
<version>2.8</version> | |
<executions> | |
<execution> |
cd /usr/hdp/current/kafka-broker/bin/ | |
[kafka@c220-node2 bin]$ ./kafka-topics.sh --create --zookeeper c220-node2.squadron-labs.com:2181 --replication-factor 2 --partitions 3 --topic source1 | |
Created topic "source1". | |
[kafka@c220-node2 bin]$ ./kafka-topics.sh --create --zookeeper c220-node2.squadron-labs.com:2181 --replication-factor 2 --partitions 3 --topic dest1 | |
Created topic "dest1". | |
[kafka@c220-node2 bin]$ ./kafka-console-producer.sh --broker-list c220-node2.squadron-labs.com:6667 --topic source1 | |
[kafka@c220-node4 bin]$ ./kafka-console-consumer.sh --bootstrap-server c220-node4.squadron-labs.com:6667 --topic dest1 | |
Kerberized Cluster
Enable hive interactive server in hive
Get following details from hive for spark or try this HWC Quick Test Script
./kafka-console-consumer.sh --bootstrap-server c320-node2.squadron-labs.com:6667 --topic __consumer_offsets --from-beginning --security-protocol SASL_PLAINTEXT --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --max-messages 10000 | cut -d, -f1 | cut -d[ -f2 | sort | uniq | |
Note: | |
1)Tune value of --max-messages to get more output,if any | |
2)Remove --security-protocol SASL_PLAINTEXT if not a secure cluster. |
bin/kafka-topics.sh --zookeeper localhost:2181 --list
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000
... wait a minute ...
A) Spark structured streaming Kafka integration - SASL_PLAINTEXT | |
1) Prerequisites | |
[consumer-user@c220-node1 sslss]$ ll | |
-rw------- 1 consumer-user root 144 Apr 21 08:56 consumer-user.keytab | |
-rw-rw-r-- 1 consumer-user consumer-user 229 Apr 21 09:40 kafka_client_jaas.conf | |
[consumer-user@c220-node1 sslss]$ cat kafka_client_jaas.conf | |
KafkaClient { |