Last active
June 5, 2024 07:32
-
-
Save asdaraujo/2c7d8c1119a45a4e7bbaa3e068655c84 to your computer and use it in GitHub Desktop.
kafka-python example with Kerberos auth
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Requirements: kafka-python gssapi krbticket | |
import os | |
import time | |
from kafka import KafkaConsumer, KafkaProducer | |
from krbticket import KrbConfig, KrbCommand | |
try: | |
os.environ['KRB5CCNAME'] = '/tmp/krb5cc_<myusername>' | |
kconfig = KrbConfig(principal='araujo', keytab='/path/to/<myusername>.keytab') | |
KrbCommand.kinit(kconfig) | |
# Kafka broker | |
BROKERS = ['host.cloudera.site:9093'] | |
# Kafka topics | |
TOPIC = 'demo' | |
producer = KafkaProducer( | |
bootstrap_servers=BROKERS, | |
security_protocol='SASL_SSL', | |
ssl_cafile='/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem', | |
sasl_mechanism='GSSAPI', | |
sasl_kerberos_service_name='kafka', | |
) | |
producer.send(TOPIC, ('Hello, World! %s' % (time.time(),)).encode()) | |
producer.flush() | |
consumer = KafkaConsumer( | |
bootstrap_servers=BROKERS, | |
security_protocol='SASL_SSL', | |
ssl_cafile='/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem', | |
sasl_mechanism='GSSAPI', | |
sasl_kerberos_service_name='kafka', | |
auto_offset_reset='earliest', | |
) | |
consumer.subscribe([TOPIC]) | |
for message in consumer: | |
print(message) | |
finally: | |
print("Destroying ticket") | |
KrbCommand.kdestroy(kconfig) |
Thanks for answering this, @Aiganymus !
Even though you don't need to set KRB5CCNAME
, I'd highly recommend you to do so. If you have multiple scripts using the default cache, the actions of a script (kinit/kdestroy) can affect the other scripts. If you set one cache file for each script, using the KRB5CCNAME
variable, then you're safe.
Hello @asdaraujo,
I'm trying to implement this to be able to connect to Kafka from a Windows Server Application which is not supported by the KrbTicket.
Do you have any idea of what could be a possible solution?
Thanks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@manishr38
what this line number 8 is meant for? - it is used to set kerberos default credential cache name
Do i have to add the environment variable? - you don't. you can check the default cache name using the command
klist
Is "KRB5CCNAME" fixed variable name? - yes
or it changes upon the server configuration? - no