Skip to content

Instantly share code, notes, and snippets.

@crcastle

crcastle/confluent.properties

Created August 11, 2016 18:13
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save crcastle/6108c429ff2c3dfb444a0b13108a3a39 to your computer and use it in GitHub Desktop.
Save crcastle/6108c429ff2c3dfb444a0b13108a3a39 to your computer and use it in GitHub Desktop.
Download ZIP
Kafka Connect with SSL Trials and Tribulations
Raw
confluent.properties
security.protocol=SSL
ssl.truststore.location=/app/.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/app/.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
bootstrap.servers=ec2-52-1-245-163.compute-1.amazonaws.com:9096,ec2-52-22-254-93.compute-1.amazonaws.com:9096,ec2-52-207-158-223.compute-1.amazonaws.com:9096
rest.port=46369
# copied from Kafka Connect distribution etc/kafka/connect-standalone.properties example file
# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
# it to
key.converter.schemas.enable=true
value.converter.schemas.enable=true
# The internal converter used for offsets and config data is configurable and must be specified, but most users will
# always want to use the built-in default. Offset and config data is never visible outside of Copcyat in this format.
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
# Flush much faster than normal, which is useful for testing/debugging
offset.flush.interval.ms=10000
Raw
startup.log
2016-08-11T18:07:02.556970+00:00 heroku[web.1]: Starting process with command `bin/run-confluent twitter-source.properties`
2016-08-11T18:07:03.197097+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2016-08-11T18:07:03.208586+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
2016-08-11T18:07:03.340574+00:00 app[web.1]: Certificate was added to keystore
2016-08-11T18:07:03.356700+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
2016-08-11T18:07:03.556107+00:00 app[web.1]: Entry for alias 1 successfully imported.
2016-08-11T18:07:03.562537+00:00 app[web.1]: Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
2016-08-11T18:07:03.588970+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
2016-08-11T18:07:03.816657+00:00 app[web.1]: SLF4J: Class path contains multiple SLF4J bindings.
2016-08-11T18:07:03.816795+00:00 app[web.1]: SLF4J: Found binding in [jar:file:/app/target/kafka-connect-twitter-0.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2016-08-11T18:07:03.816851+00:00 app[web.1]: SLF4J: Found binding in [jar:file:/app/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2016-08-11T18:07:03.816905+00:00 app[web.1]: SLF4J: Found binding in [jar:file:/app/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2016-08-11T18:07:03.816954+00:00 app[web.1]: SLF4J: Found binding in [jar:file:/app/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2016-08-11T18:07:03.817002+00:00 app[web.1]: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2016-08-11T18:07:03.818984+00:00 app[web.1]: SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
2016-08-11T18:07:03.848071+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.standalone.StandaloneConfig - StandaloneConfig values:
2016-08-11T18:07:03.848076+00:00 app[web.1]: cluster = connect
2016-08-11T18:07:03.848077+00:00 app[web.1]: rest.advertised.host.name = null
2016-08-11T18:07:03.848078+00:00 app[web.1]: task.shutdown.graceful.timeout.ms = 5000
2016-08-11T18:07:03.848079+00:00 app[web.1]: rest.host.name = null
2016-08-11T18:07:03.848079+00:00 app[web.1]: rest.advertised.port = null
2016-08-11T18:07:03.848084+00:00 app[web.1]: bootstrap.servers = [ec2-52-1-245-163.compute-1.amazonaws.com:9096, ec2-52-22-254-93.compute-1.amazonaws.com:9096, ec2-52-207-158-223.compute-1.amazonaws.com:9096]
2016-08-11T18:07:03.848085+00:00 app[web.1]: offset.flush.timeout.ms = 5000
2016-08-11T18:07:03.848086+00:00 app[web.1]: offset.flush.interval.ms = 10000
2016-08-11T18:07:03.848086+00:00 app[web.1]: rest.port = 23779
2016-08-11T18:07:03.848087+00:00 app[web.1]: internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
2016-08-11T18:07:03.848088+00:00 app[web.1]: access.control.allow.methods =
2016-08-11T18:07:03.848095+00:00 app[web.1]: access.control.allow.origin =
2016-08-11T18:07:03.848096+00:00 app[web.1]: offset.storage.file.filename = /tmp/connect.offsets
2016-08-11T18:07:03.848097+00:00 app[web.1]: internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
2016-08-11T18:07:03.848098+00:00 app[web.1]: value.converter = class org.apache.kafka.connect.storage.StringConverter
2016-08-11T18:07:03.848098+00:00 app[web.1]: key.converter = class org.apache.kafka.connect.storage.StringConverter
2016-08-11T18:07:03.848106+00:00 app[web.1]:
2016-08-11T18:07:03.989290+00:00 app[web.1]: [main] INFO org.eclipse.jetty.util.log - Logging initialized @397ms
2016-08-11T18:07:04.173453+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Connect - Kafka Connect starting
2016-08-11T18:07:04.173549+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.standalone.StandaloneHerder - Herder starting
2016-08-11T18:07:04.173624+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Worker starting
2016-08-11T18:07:04.182281+00:00 app[web.1]: [main] INFO org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values:
2016-08-11T18:07:04.182284+00:00 app[web.1]: metric.reporters = []
2016-08-11T18:07:04.182284+00:00 app[web.1]: metadata.max.age.ms = 300000
2016-08-11T18:07:04.182285+00:00 app[web.1]: reconnect.backoff.ms = 50
2016-08-11T18:07:04.182286+00:00 app[web.1]: sasl.kerberos.ticket.renew.window.factor = 0.8
2016-08-11T18:07:04.182287+00:00 app[web.1]: bootstrap.servers = [ec2-52-1-245-163.compute-1.amazonaws.com:9096, ec2-52-22-254-93.compute-1.amazonaws.com:9096, ec2-52-207-158-223.compute-1.amazonaws.com:9096]
2016-08-11T18:07:04.182287+00:00 app[web.1]: ssl.keystore.type = JKS
2016-08-11T18:07:04.182288+00:00 app[web.1]: sasl.mechanism = GSSAPI
2016-08-11T18:07:04.182289+00:00 app[web.1]: max.block.ms = 9223372036854775807
2016-08-11T18:07:04.182289+00:00 app[web.1]: interceptor.classes = null
2016-08-11T18:07:04.182290+00:00 app[web.1]: ssl.truststore.password = null
2016-08-11T18:07:04.182290+00:00 app[web.1]: client.id =
2016-08-11T18:07:04.182291+00:00 app[web.1]: ssl.endpoint.identification.algorithm = null
2016-08-11T18:07:04.182291+00:00 app[web.1]: request.timeout.ms = 2147483647
2016-08-11T18:07:04.182292+00:00 app[web.1]: acks = all
2016-08-11T18:07:04.182292+00:00 app[web.1]: receive.buffer.bytes = 32768
2016-08-11T18:07:04.182293+00:00 app[web.1]: ssl.truststore.type = JKS
2016-08-11T18:07:04.182293+00:00 app[web.1]: retries = 2147483647
2016-08-11T18:07:04.182294+00:00 app[web.1]: ssl.truststore.location = null
2016-08-11T18:07:04.182294+00:00 app[web.1]: ssl.keystore.password = null
2016-08-11T18:07:04.182295+00:00 app[web.1]: send.buffer.bytes = 131072
2016-08-11T18:07:04.182295+00:00 app[web.1]: compression.type = none
2016-08-11T18:07:04.182296+00:00 app[web.1]: metadata.fetch.timeout.ms = 60000
2016-08-11T18:07:04.182296+00:00 app[web.1]: retry.backoff.ms = 100
2016-08-11T18:07:04.182297+00:00 app[web.1]: sasl.kerberos.kinit.cmd = /usr/bin/kinit
2016-08-11T18:07:04.182297+00:00 app[web.1]: buffer.memory = 33554432
2016-08-11T18:07:04.182298+00:00 app[web.1]: timeout.ms = 30000
2016-08-11T18:07:04.182299+00:00 app[web.1]: key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
2016-08-11T18:07:04.182299+00:00 app[web.1]: sasl.kerberos.service.name = null
2016-08-11T18:07:04.182300+00:00 app[web.1]: sasl.kerberos.ticket.renew.jitter = 0.05
2016-08-11T18:07:04.182300+00:00 app[web.1]: ssl.trustmanager.algorithm = PKIX
2016-08-11T18:07:04.182301+00:00 app[web.1]: block.on.buffer.full = false
2016-08-11T18:07:04.182301+00:00 app[web.1]: ssl.key.password = null
2016-08-11T18:07:04.182302+00:00 app[web.1]: sasl.kerberos.min.time.before.relogin = 60000
2016-08-11T18:07:04.182303+00:00 app[web.1]: connections.max.idle.ms = 540000
2016-08-11T18:07:04.182303+00:00 app[web.1]: max.in.flight.requests.per.connection = 1
2016-08-11T18:07:04.182304+00:00 app[web.1]: metrics.num.samples = 2
2016-08-11T18:07:04.182304+00:00 app[web.1]: ssl.protocol = TLS
2016-08-11T18:07:04.182305+00:00 app[web.1]: ssl.provider = null
2016-08-11T18:07:04.182306+00:00 app[web.1]: ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
2016-08-11T18:07:04.182306+00:00 app[web.1]: batch.size = 16384
2016-08-11T18:07:04.182306+00:00 app[web.1]: ssl.keystore.location = null
2016-08-11T18:07:04.182307+00:00 app[web.1]: ssl.cipher.suites = null
2016-08-11T18:07:04.182307+00:00 app[web.1]: security.protocol = PLAINTEXT
2016-08-11T18:07:04.182308+00:00 app[web.1]: max.request.size = 1048576
2016-08-11T18:07:04.182309+00:00 app[web.1]: value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
2016-08-11T18:07:04.182309+00:00 app[web.1]: ssl.keymanager.algorithm = SunX509
2016-08-11T18:07:04.182310+00:00 app[web.1]: metrics.sample.window.ms = 30000
2016-08-11T18:07:04.182310+00:00 app[web.1]: partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
2016-08-11T18:07:04.182311+00:00 app[web.1]: linger.ms = 0
2016-08-11T18:07:04.182311+00:00 app[web.1]:
2016-08-11T18:07:04.213977+00:00 app[web.1]: [main] INFO org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values:
2016-08-11T18:07:04.213981+00:00 app[web.1]: metric.reporters = []
2016-08-11T18:07:04.213982+00:00 app[web.1]: metadata.max.age.ms = 300000
2016-08-11T18:07:04.213982+00:00 app[web.1]: reconnect.backoff.ms = 50
2016-08-11T18:07:04.213983+00:00 app[web.1]: sasl.kerberos.ticket.renew.window.factor = 0.8
2016-08-11T18:07:04.213983+00:00 app[web.1]: bootstrap.servers = [ec2-52-1-245-163.compute-1.amazonaws.com:9096, ec2-52-22-254-93.compute-1.amazonaws.com:9096, ec2-52-207-158-223.compute-1.amazonaws.com:9096]
2016-08-11T18:07:04.213984+00:00 app[web.1]: ssl.keystore.type = JKS
2016-08-11T18:07:04.213984+00:00 app[web.1]: sasl.mechanism = GSSAPI
2016-08-11T18:07:04.213985+00:00 app[web.1]: max.block.ms = 9223372036854775807
2016-08-11T18:07:04.213985+00:00 app[web.1]: interceptor.classes = null
2016-08-11T18:07:04.213985+00:00 app[web.1]: ssl.truststore.password = null
2016-08-11T18:07:04.213986+00:00 app[web.1]: client.id = producer-1
2016-08-11T18:07:04.213986+00:00 app[web.1]: ssl.endpoint.identification.algorithm = null
2016-08-11T18:07:04.213990+00:00 app[web.1]: request.timeout.ms = 2147483647
2016-08-11T18:07:04.213991+00:00 app[web.1]: acks = all
2016-08-11T18:07:04.213991+00:00 app[web.1]: receive.buffer.bytes = 32768
2016-08-11T18:07:04.213992+00:00 app[web.1]: ssl.truststore.type = JKS
2016-08-11T18:07:04.213992+00:00 app[web.1]: retries = 2147483647
2016-08-11T18:07:04.213992+00:00 app[web.1]: ssl.truststore.location = null
2016-08-11T18:07:04.213992+00:00 app[web.1]: ssl.keystore.password = null
2016-08-11T18:07:04.213993+00:00 app[web.1]: send.buffer.bytes = 131072
2016-08-11T18:07:04.213993+00:00 app[web.1]: compression.type = none
2016-08-11T18:07:04.213994+00:00 app[web.1]: metadata.fetch.timeout.ms = 60000
2016-08-11T18:07:04.213994+00:00 app[web.1]: retry.backoff.ms = 100
2016-08-11T18:07:04.213994+00:00 app[web.1]: sasl.kerberos.kinit.cmd = /usr/bin/kinit
2016-08-11T18:07:04.213995+00:00 app[web.1]: buffer.memory = 33554432
2016-08-11T18:07:04.213995+00:00 app[web.1]: timeout.ms = 30000
2016-08-11T18:07:04.213996+00:00 app[web.1]: key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
2016-08-11T18:07:04.213996+00:00 app[web.1]: sasl.kerberos.service.name = null
2016-08-11T18:07:04.213997+00:00 app[web.1]: sasl.kerberos.ticket.renew.jitter = 0.05
2016-08-11T18:07:04.213997+00:00 app[web.1]: ssl.trustmanager.algorithm = PKIX
2016-08-11T18:07:04.213997+00:00 app[web.1]: block.on.buffer.full = false
2016-08-11T18:07:04.213998+00:00 app[web.1]: ssl.key.password = null
2016-08-11T18:07:04.213998+00:00 app[web.1]: sasl.kerberos.min.time.before.relogin = 60000
2016-08-11T18:07:04.213998+00:00 app[web.1]: connections.max.idle.ms = 540000
2016-08-11T18:07:04.213999+00:00 app[web.1]: max.in.flight.requests.per.connection = 1
2016-08-11T18:07:04.213999+00:00 app[web.1]: metrics.num.samples = 2
2016-08-11T18:07:04.214000+00:00 app[web.1]: ssl.protocol = TLS
2016-08-11T18:07:04.214000+00:00 app[web.1]: ssl.provider = null
2016-08-11T18:07:04.214001+00:00 app[web.1]: ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
2016-08-11T18:07:04.214001+00:00 app[web.1]: batch.size = 16384
2016-08-11T18:07:04.214002+00:00 app[web.1]: ssl.keystore.location = null
2016-08-11T18:07:04.214002+00:00 app[web.1]: ssl.cipher.suites = null
2016-08-11T18:07:04.214002+00:00 app[web.1]: security.protocol = PLAINTEXT
2016-08-11T18:07:04.214003+00:00 app[web.1]: max.request.size = 1048576
2016-08-11T18:07:04.214003+00:00 app[web.1]: value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
2016-08-11T18:07:04.214004+00:00 app[web.1]: ssl.keymanager.algorithm = SunX509
2016-08-11T18:07:04.214004+00:00 app[web.1]: metrics.sample.window.ms = 30000
2016-08-11T18:07:04.214004+00:00 app[web.1]: partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
2016-08-11T18:07:04.214005+00:00 app[web.1]: linger.ms = 0
2016-08-11T18:07:04.214010+00:00 app[web.1]:
2016-08-11T18:07:04.215308+00:00 app[web.1]: [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.10.0.0
2016-08-11T18:07:04.215365+00:00 app[web.1]: [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : b8642491e78c5a13
2016-08-11T18:07:04.216256+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.storage.FileOffsetBackingStore - Starting FileOffsetBackingStore with file /tmp/connect.offsets
2016-08-11T18:07:04.263509+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Worker started
2016-08-11T18:07:04.263889+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.standalone.StandaloneHerder - Herder started
2016-08-11T18:07:04.263944+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.rest.RestServer - Starting REST server
2016-08-11T18:07:04.346564+00:00 app[web.1]: [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.15.v20160210
2016-08-11T18:07:04.890368+00:00 app[web.1]: Aug 11, 2016 6:07:04 PM org.glassfish.jersey.internal.Errors logErrors
2016-08-11T18:07:04.890409+00:00 app[web.1]: WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
2016-08-11T18:07:04.890411+00:00 app[web.1]: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
2016-08-11T18:07:04.890412+00:00 app[web.1]: WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
2016-08-11T18:07:04.890412+00:00 app[web.1]: WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
2016-08-11T18:07:04.890413+00:00 app[web.1]:
2016-08-11T18:07:04.892708+00:00 app[web.1]: [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@14dda234{/,null,AVAILABLE}
2016-08-11T18:07:04.900598+00:00 app[web.1]: [main] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector@f478a81{HTTP/1.1}{0.0.0.0:23779}
2016-08-11T18:07:04.900814+00:00 app[web.1]: [main] INFO org.eclipse.jetty.server.Server - Started @1311ms
2016-08-11T18:07:04.901194+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.rest.RestServer - REST server listening at http://172.18.44.130:23779/, advertising URL http://172.18.44.130:23779/
2016-08-11T18:07:04.901259+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Connect - Kafka Connect started
2016-08-11T18:07:04.902059+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.ConnectorConfig - ConnectorConfig values:
2016-08-11T18:07:04.902061+00:00 app[web.1]: connector.class = com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
2016-08-11T18:07:04.902062+00:00 app[web.1]: tasks.max = 1
2016-08-11T18:07:04.902063+00:00 app[web.1]: name = twitter-source
2016-08-11T18:07:04.902066+00:00 app[web.1]:
2016-08-11T18:07:04.908126+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Creating connector twitter-source of type com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
2016-08-11T18:07:04.909261+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Instantiated connector twitter-source with version of type com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
2016-08-11T18:07:05.032432+00:00 app[web.1]: [main] INFO com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector - Starting Twitter source task with {connector.class=com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector, twitter.token=3259381-9FAZeW4qkEyhYAkMh4pK0royzAxnJJ39LBlaGkuMAM, output.format=string, tasks.max=1, twitter.secret=[hidden], track.terms=news,music,hadoop,clojure,scala,fp,golang,python,fsharp,cpp,java, stream.type=filter, name=twitter-source, topic=test, twitter.consumersecret=[hidden], language=en, twitter.consumerkey=1HMhyzB4311tznCwc6NPEsXcN}.
2016-08-11T18:07:05.035440+00:00 app[web.1]: [main] INFO com.eneco.trading.kafka.connect.twitter.TwitterSourceConfig - TwitterSourceConfig values:
2016-08-11T18:07:05.035442+00:00 app[web.1]: twitter.token = 3259381-9FAZeW4qkEyhYAkMh4pK0royzAxnJJ39LBlaGkuMAM
2016-08-11T18:07:05.035443+00:00 app[web.1]: twitter.secret = [hidden]
2016-08-11T18:07:05.035444+00:00 app[web.1]: stream.type = filter
2016-08-11T18:07:05.035445+00:00 app[web.1]: track.terms = [news, music, hadoop, clojure, scala, fp, golang, python, fsharp, cpp, java]
2016-08-11T18:07:05.035446+00:00 app[web.1]: batch.size = 100
2016-08-11T18:07:05.035447+00:00 app[web.1]: language = [en]
2016-08-11T18:07:05.035447+00:00 app[web.1]: track.follow = []
2016-08-11T18:07:05.035448+00:00 app[web.1]: track.locations = []
2016-08-11T18:07:05.035448+00:00 app[web.1]: output.format = string
2016-08-11T18:07:05.035449+00:00 app[web.1]: topic = test
2016-08-11T18:07:05.035450+00:00 app[web.1]: twitter.consumersecret = [hidden]
2016-08-11T18:07:05.035451+00:00 app[web.1]: twitter.consumerkey = 1HMhyzB4311tznCwc6NPEsXcN
2016-08-11T18:07:05.035452+00:00 app[web.1]: twitter.app.name = KafkaConnectTwitterSource
2016-08-11T18:07:05.035452+00:00 app[web.1]: batch.timeout = 0.1
2016-08-11T18:07:05.035455+00:00 app[web.1]:
2016-08-11T18:07:05.047575+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Finished creating connector twitter-source
2016-08-11T18:07:05.047924+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.SourceConnectorConfig - SourceConnectorConfig values:
2016-08-11T18:07:05.047926+00:00 app[web.1]: connector.class = com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
2016-08-11T18:07:05.047928+00:00 app[web.1]: tasks.max = 1
2016-08-11T18:07:05.047929+00:00 app[web.1]: name = twitter-source
2016-08-11T18:07:05.047931+00:00 app[web.1]:
2016-08-11T18:07:05.048544+00:00 app[web.1]: [main] INFO com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector - Setting task configurations for 1 workers.
2016-08-11T18:07:05.057757+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.TaskConfig - TaskConfig values:
2016-08-11T18:07:05.057760+00:00 app[web.1]: task.class = class com.eneco.trading.kafka.connect.twitter.TwitterSourceTask
2016-08-11T18:07:05.057764+00:00 app[web.1]:
2016-08-11T18:07:05.057865+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Creating task twitter-source-0
2016-08-11T18:07:05.058044+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.runtime.Worker - Instantiated task twitter-source-0 with version of type com.eneco.trading.kafka.connect.twitter.TwitterSourceTask
2016-08-11T18:07:05.061275+00:00 app[web.1]: [main] INFO org.apache.kafka.connect.cli.ConnectStandalone - Created connector twitter-source
2016-08-11T18:07:05.061928+00:00 app[web.1]: [pool-2-thread-1] INFO com.eneco.trading.kafka.connect.twitter.TwitterSourceConfig - TwitterSourceConfig values:
2016-08-11T18:07:05.061930+00:00 app[web.1]: twitter.token = 3259381-9FAZeW4qkEyhYAkMh4pK0royzAxnJJ39LBlaGkuMAM
2016-08-11T18:07:05.061931+00:00 app[web.1]: twitter.secret = [hidden]
2016-08-11T18:07:05.061931+00:00 app[web.1]: stream.type = filter
2016-08-11T18:07:05.061932+00:00 app[web.1]: track.terms = [news, music, hadoop, clojure, scala, fp, golang, python, fsharp, cpp, java]
2016-08-11T18:07:05.061933+00:00 app[web.1]: batch.size = 100
2016-08-11T18:07:05.061934+00:00 app[web.1]: language = [en]
2016-08-11T18:07:05.061934+00:00 app[web.1]: track.follow = []
2016-08-11T18:07:05.061935+00:00 app[web.1]: track.locations = []
2016-08-11T18:07:05.061935+00:00 app[web.1]: output.format = string
2016-08-11T18:07:05.061936+00:00 app[web.1]: topic = test
2016-08-11T18:07:05.061937+00:00 app[web.1]: twitter.consumersecret = [hidden]
2016-08-11T18:07:05.061937+00:00 app[web.1]: twitter.consumerkey = 1HMhyzB4311tznCwc6NPEsXcN
2016-08-11T18:07:05.061938+00:00 app[web.1]: twitter.app.name = KafkaConnectTwitterSource
2016-08-11T18:07:05.061938+00:00 app[web.1]: batch.timeout = 0.1
2016-08-11T18:07:05.061941+00:00 app[web.1]:
2016-08-11T18:07:05.231374+00:00 app[web.1]: [pool-2-thread-1] INFO com.eneco.trading.kafka.connect.twitter.TwitterStatusReader - Initialising Twitter Stream Reader
2016-08-11T18:07:05.246281+00:00 app[web.1]: [pool-2-thread-1] INFO com.twitter.hbc.httpclient.BasicClient - New connection executed: KafkaConnectTwitterSource, endpoint: /1.1/statuses/filter.json?delimited=length&language=en
2016-08-11T18:07:05.246821+00:00 app[web.1]: [pool-2-thread-1] INFO org.apache.kafka.connect.runtime.WorkerSourceTask - Source task WorkerSourceTask{id=twitter-source-0} finished initialization and start
2016-08-11T18:07:05.343975+00:00 app[web.1]: [hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - KafkaConnectTwitterSource Establishing a connection
2016-08-11T18:07:05.160369+00:00 heroku[web.1]: State changed from starting to up
2016-08-11T18:07:05.759533+00:00 app[web.1]: [hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - KafkaConnectTwitterSource Processing connection data
2016-08-11T18:07:06.080155+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.277327+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.377396+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.522903+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.687509+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.733025+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:06.824062+00:00 app[web.1]: [pool-7-thread-1] INFO com.eneco.trading.kafka.connect.twitter.StatusEnqueuer - onTrackLimitationNotice 5
2016-08-11T18:07:06.880022+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.076711+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.309065+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.312141+00:00 app[web.1]: [pool-7-thread-1] INFO com.eneco.trading.kafka.connect.twitter.StatusEnqueuer - onTrackLimitationNotice 4
2016-08-11T18:07:07.468657+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.514533+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.656249+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.801676+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:07.938564+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:08.161257+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:08.286479+00:00 app[web.1]: [pool-7-thread-1] INFO com.eneco.trading.kafka.connect.twitter.StatusEnqueuer - onTrackLimitationNotice 1
2016-08-11T18:07:08.454146+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:08.457680+00:00 app[web.1]: [CLASSPATH traversal thread.] INFO org.reflections.Reflections - Reflections took 4155 ms to scan 122 urls, producing 4852 keys and 36107 values
2016-08-11T18:07:08.707494+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:08.886719+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.144429+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.167401+00:00 app[web.1]: [pool-7-thread-1] INFO com.eneco.trading.kafka.connect.twitter.StatusEnqueuer - onTrackLimitationNotice 7
2016-08-11T18:07:09.324792+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.468207+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.610482+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.653530+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.809417+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:09.951956+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.095114+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.237884+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.279325+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.426961+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.571477+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.714734+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.858723+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:10.900828+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.047404+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.200112+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.241795+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.386263+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.528639+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.586615+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.730946+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.772829+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:11.917378+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.059161+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.204218+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.347484+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.390581+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.533141+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.676555+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.820037+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:12.974560+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.117029+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.260309+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.302186+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.462219+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.604598+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.747567+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.890078+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:13.931979+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:14.075796+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-207-158-223.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:14.220405+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-22-254-93.compute-1.amazonaws.com:9096 disconnected
2016-08-11T18:07:14.361466+00:00 app[web.1]: [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - Bootstrap broker ec2-52-1-245-163.compute-1.amazonaws.com:9096 disconnected
^C
Raw
twitter-source.properties
name=twitter-source
connector.class=com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
tasks.max=1
topic=test
twitter.consumerkey=1HMhyzB4311tznCwc6NPEsXcN
twitter.consumersecret=[hidden]
twitter.token=3259381-9FAZeW4qkEyhYAkMh4pK0royzAxnJJ39LBlaGkuMAM
twitter.secret=[hidden]
# set output.format to string to output string key/values, it defaults to structured
output.format=string
language=en
#stream.type=sample
stream.type=filter
track.terms=news,music,hadoop,clojure,scala,fp,golang,python,fsharp,cpp,java
# San Francisco OR New York City
#track.locations=-122.75,36.8,-121.75,37.8,-74,40,-73,41
# bbcbreaking,bbcnews,justinbieber
# track.follow=5402612,612473,27260086
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment