Skip to content

Instantly share code, notes, and snippets.

@kishorenc

kishorenc/gist:ef806e85478378ce2203

Created December 15, 2015 13:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kishorenc/ef806e85478378ce2203 to your computer and use it in GitHub Desktop.
Save kishorenc/ef806e85478378ce2203 to your computer and use it in GitHub Desktop.
Download ZIP
Samza error log 2
Raw
gistfile1.txt
2015-12-15 11:50:14 SamzaContainer$ [INFO] Got container ID: 59
2015-12-15 11:50:14 SamzaContainer$ [INFO] Got coordinator URL: http://10.181.18.44:39659/
2015-12-15 11:50:14 SamzaContainer$ [INFO] Fetching configuration from: http://10.181.18.44:39659/
2015-12-15 11:50:18 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are c1-yarn-i-9f717e2f.stag.company.io
2015-12-15 11:50:18 JmxServer [INFO] Started JmxServer registry port=54113 server port=54114 url=service:jmx:rmi://localhost:54114/jndi/rmi://localhost:54113/jmxrmi
2015-12-15 11:50:18 JmxServer [INFO] If you are tunneling, you might want to try JmxServer registry port=54113 server port=54114 url=service:jmx:rmi://c1-yarn-i-9f717e2f.stag.company.io:54114/jndi/rmi://c1-yarn-i-9f717e2f.stag.company.io:54113/jmxrmi
2015-12-15 11:50:18 SamzaContainer$ [INFO] Setting up Samza container: samza-container-59
2015-12-15 11:50:18 SamzaContainer$ [INFO] Samza container PID: 8596@c1-yarn-i-9f717e2f.stag.company.io
2015-12-15 11:50:18 SamzaContainer$ [INFO] Using configuration: {stores.kvstore.changelog.replication.factor=2, stores.kvstore.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory, stores.kvstore.container.cache.size.bytes=8000000, systems.kafka.streams.TOPIC_ONE.samza.msg.serde=bytes, stores.kvstore.rocksdb.write.batch.size=2000, stores.metastore.msg.serde=object, stores.kvstore.rocksdb.compression=snappy, systems.kafka.streams.SamzaJob.samza.msg.serde=bytes, systems.kafka.streams.TOPIC_TWO.samza.key.serde=string, serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory, stores.kvstore.rocksdb.object.cache.size=2000, systems.kafka.streams.TOPIC_ONE.samza.offset.default=oldest, task.window.ms=50000, serializers.registry.object.class=org.apache.samza.serializers.SerializableSerdeFactory, systems.kafka.streams.SamzaJob.samza.key.serde=string, task.class=spike.streaming.samza.PricesWithLatestUrl, task.commit.ms=60000, yarn.package.path=http://yarn01.stag.company.io:8000/samza_job.tar.gz, systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory, stores.metastore.factory=org.apache.samza.storage.kv.inmemory.InMemoryKeyValueStorageEngineFactory, task.inputs=kafka.TOPIC_TWO,kafka.TOPIC_ONE, yarn.container.memory.mb=512, job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, stores.metastore.changelog=kafka.samza-job-meta-changelog, task.checkpoint.replication.factor=2, serializers.registry.bytes.class=org.apache.samza.serializers.ByteSerdeFactory, systems.kafka.streams.TOPIC_TWO.samza.msg.serde=bytes, systems.kafka.streams.TOPIC_ONE.samza.key.serde=string, job.name=samza-job-production-1, stores.kvstore.rocksdb.compaction.style=level, systems.kafka.consumer.zookeeper.connect=10.181.18.223:2181,10.181.18.194:2181,10.181.18.171:2181,10.181.18.244:2181,10.181.18.134:2181,10.181.18.87:2181,10.181.18.193:2181, task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory, systems.kafka.streams.TOPIC_TWO.samza.offset.default=oldest, stores.kvstore.key.serde=bytes, systems.kafka.streams.SamzaJob.samza.offset.default=oldest, stores.metastore.changelog.replication.factor=2, yarn.container.count=200, stores.kvstore.msg.serde=object, systems.kafka.producer.bootstrap.servers=10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082, task.checkpoint.system=kafka, stores.kvstore.rocksdb.block.size.bytes=262144, task.log4j.system=kafka, stores.kvstore.rocksdb.container.write.buffer.size.bytes=134217728, stores.metastore.key.serde=string, stores.kvstore.changelog=kafka.samza-job-changelog}
2015-12-15 11:50:18 SamzaContainer$ [INFO] Using container model: ContainerModel [containerId=59, tasks={Partition 151=TaskModel [taskName=Partition 151, systemStreamPartitions=[SystemStreamPartition [kafka, TOPIC_ONE, 151], SystemStreamPartition [kafka, TOPIC_TWO, 151]], changeLogPartition=Partition [partition=202]], Partition 331=TaskModel [taskName=Partition 331, systemStreamPartitions=[SystemStreamPartition [kafka, TOPIC_ONE, 331], SystemStreamPartition [kafka, TOPIC_TWO, 331]], changeLogPartition=Partition [partition=242]]}]
2015-12-15 11:50:18 SamzaContainer$ [INFO] Got system names: Set(kafka)
2015-12-15 11:50:18 SamzaContainer$ [INFO] Got serde names: Set(string, bytes, object)
2015-12-15 11:50:18 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:50:18 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:50:18 VerifiableProperties [INFO] Property group.id is overridden to undefined-samza-consumer-group-94603aa9-7afb-4b6a-b752-14811622cc0d
2015-12-15 11:50:18 VerifiableProperties [INFO] Property zookeeper.connect is overridden to 10.181.18.223:2181,10.181.18.194:2181,10.181.18.171:2181,10.181.18.244:2181,10.181.18.134:2181,10.181.18.87:2181,10.181.18.193:2181
2015-12-15 11:50:18 KafkaSystemFactory [INFO] Creating topic meta information for topic: samza-job-meta-changelog with replication factor: 2
2015-12-15 11:50:18 KafkaSystemFactory [INFO] Creating topic meta information for topic: samza-job-changelog with replication factor: 2
2015-12-15 11:50:18 SamzaContainer$ [INFO] Got system factories: Set(kafka)
2015-12-15 11:50:18 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:50:18 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:50:18 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:50:18 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:50:18 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:50:18 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 11:50:18 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 11:50:48 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:51:18 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:51:19 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:51:19 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:51:19 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:51:19 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:51:19 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:51:19 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:51:20 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:51:50 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:52:20 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:52:21 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:52:21 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:52:21 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:52:21 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:52:21 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:52:21 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:52:22 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:52:52 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:53:22 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:53:24 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:53:24 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:53:24 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:53:24 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:53:24 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:53:54 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 11:53:54 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:53:54 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 11:53:54 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:53:54 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:53:54 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:53:54 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:54:25 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:54:55 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:54:59 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:54:59 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:54:59 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:54:59 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:54:59 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:55:29 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 11:55:29 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:55:29 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 11:55:29 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:55:29 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:55:29 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:55:29 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:55:59 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:56:29 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:56:37 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:56:37 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:56:37 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:56:37 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:56:37 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:56:37 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:56:37 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:57:07 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:57:37 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:57:47 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:57:47 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:57:47 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:57:47 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:57:47 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:57:47 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 11:57:47 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 11:58:17 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 11:58:47 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 11:58:57 VerifiableProperties [INFO] Verifying properties
2015-12-15 11:58:57 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 11:58:57 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 11:58:57 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 11:58:57 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:59:27 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 11:59:27 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:59:27 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 11:59:27 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 11:59:27 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 11:59:27 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 11:59:28 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 11:59:58 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:00:28 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:00:38 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:00:38 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:00:38 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:00:38 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:00:38 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:01:08 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:01:08 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:01:08 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:01:08 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:01:08 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:01:08 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:01:08 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:01:38 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:02:08 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:02:18 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:02:18 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:02:18 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:02:18 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:02:18 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:02:18 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:02:18 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:02:48 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:03:18 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:03:28 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:03:28 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:03:28 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:03:28 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:03:28 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:03:28 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:03:28 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:03:58 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:04:28 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:04:38 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:04:38 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:04:38 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:04:38 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:04:38 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:04:38 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:04:38 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:05:08 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:05:38 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:05:48 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:05:48 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:05:48 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:05:48 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:05:48 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:05:48 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:05:48 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:06:18 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:06:48 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:06:58 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:06:58 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:06:58 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:06:58 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:06:58 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:06:58 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:06:59 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:07:29 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:07:59 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:08:09 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:08:09 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:08:09 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:08:09 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:08:09 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:08:09 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:08:09 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:08:39 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:09:09 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:09:19 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:09:19 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:09:19 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:09:19 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:09:19 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:09:19 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:09:19 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:09:49 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:10:19 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:10:29 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:10:29 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:10:29 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:10:29 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:10:29 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:10:59 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:10:59 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:10:59 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:10:59 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:10:59 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:10:59 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:10:59 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:11:30 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:12:00 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:12:10 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:12:10 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:12:10 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:12:10 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:12:10 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:12:40 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:12:40 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:12:40 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:12:40 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:12:40 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:12:40 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:12:40 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:13:10 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:13:40 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:13:50 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:13:50 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:13:50 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:13:50 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:13:50 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:13:50 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:13:50 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:14:20 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:14:50 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:15:00 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:15:00 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:15:00 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:15:00 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:15:00 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:15:30 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:15:30 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:15:30 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:15:30 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:15:30 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:15:30 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:15:30 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:16:00 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:16:30 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:16:40 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:16:40 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:16:40 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:16:40 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:16:40 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:16:40 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:16:40 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:17:10 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:17:40 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:17:50 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:17:50 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:17:50 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:17:50 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:17:50 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:18:20 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:18:20 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:18:20 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:18:20 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:18:20 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:18:20 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:18:20 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:18:50 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:19:20 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:19:30 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:19:30 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:19:30 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:19:30 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:19:30 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:19:30 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:19:31 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:20:01 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:20:31 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:20:41 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:20:41 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:20:41 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:20:41 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:20:41 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:20:41 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:20:42 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:21:12 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:21:42 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:21:52 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:21:52 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:21:52 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:21:52 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:21:52 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:21:52 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:21:52 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:22:22 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:22:52 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:23:02 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:23:02 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:23:02 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:23:02 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:23:02 ClientUtils$ [INFO] Fetching metadata from broker id:3,host:10.181.18.244,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:23:02 SyncProducer [INFO] Connected to 10.181.18.244:9082 for producing
2015-12-15 12:23:02 SyncProducer [INFO] Disconnecting from 10.181.18.244:9082
2015-12-15 12:23:32 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:24:02 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:24:12 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:24:12 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:24:12 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:24:12 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:24:12 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:24:12 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:24:12 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:24:42 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:25:12 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:25:22 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:25:22 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:25:22 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:25:22 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:25:22 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:25:22 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:25:22 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:25:52 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:26:22 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:26:32 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:26:32 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:26:32 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:26:32 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:26:32 ClientUtils$ [INFO] Fetching metadata from broker id:3,host:10.181.18.244,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:26:32 SyncProducer [INFO] Connected to 10.181.18.244:9082 for producing
2015-12-15 12:26:32 SyncProducer [INFO] Disconnecting from 10.181.18.244:9082
2015-12-15 12:27:02 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:27:32 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:27:42 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:27:42 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:27:42 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:27:42 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:27:42 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:27:42 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:27:42 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:28:12 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:28:42 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:28:52 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:28:52 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:28:52 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:28:52 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:28:52 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:28:52 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:28:52 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:29:22 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:29:52 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:30:02 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:30:02 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:30:02 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:30:02 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:30:02 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:30:02 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:30:03 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:30:33 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:31:03 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:31:13 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:31:13 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:31:13 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:31:13 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:31:13 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:31:13 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:31:13 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:31:43 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:32:13 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:32:23 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:32:23 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:32:23 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:32:23 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:32:23 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:32:23 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:32:23 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:32:53 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:33:23 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:33:33 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:33:33 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:33:33 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:33:33 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:33:33 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:33:33 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:33:35 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:34:05 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:34:35 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:34:45 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:34:45 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:34:45 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:34:45 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:34:45 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:34:45 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:34:46 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:35:16 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:35:46 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:35:56 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:35:56 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:35:56 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:35:56 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:35:56 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:35:56 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:35:56 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:36:26 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:36:56 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:37:06 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:37:06 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:37:06 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:37:06 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:37:06 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:37:06 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:37:06 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:37:36 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:38:06 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:38:16 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:38:16 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:38:16 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:38:16 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:38:16 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:38:16 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:38:33 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:39:03 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:39:33 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:39:43 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:39:43 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:39:43 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:39:43 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:39:43 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:39:43 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:39:43 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:40:13 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:40:43 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:40:53 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:40:53 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:40:53 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:40:53 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:40:53 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:40:53 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:40:53 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:41:23 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:41:53 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:42:03 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:42:03 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:42:03 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:42:03 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:42:03 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:42:03 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:42:03 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:42:33 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:43:03 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:43:13 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:43:13 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:43:13 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:43:13 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:43:13 ClientUtils$ [INFO] Fetching metadata from broker id:6,host:10.181.18.193,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:43:13 SyncProducer [INFO] Connected to 10.181.18.193:9082 for producing
2015-12-15 12:43:13 SyncProducer [INFO] Disconnecting from 10.181.18.193:9082
2015-12-15 12:43:43 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:44:13 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:44:23 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:44:23 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:44:23 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:44:23 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:44:23 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:44:23 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:44:23 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:44:53 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:45:23 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:45:33 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:45:33 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:45:33 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:45:33 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:45:33 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:46:03 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:46:03 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:46:03 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:46:03 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:46:03 ClientUtils$ [INFO] Fetching metadata from broker id:4,host:10.181.18.134,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:46:03 SyncProducer [INFO] Connected to 10.181.18.134:9082 for producing
2015-12-15 12:46:04 SyncProducer [INFO] Disconnecting from 10.181.18.134:9082
2015-12-15 12:46:34 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:47:04 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:47:14 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:47:14 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:47:14 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:47:14 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:47:14 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:47:14 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:47:14 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:47:44 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:48:14 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:48:24 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:48:24 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:48:24 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:48:24 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:48:24 ClientUtils$ [INFO] Fetching metadata from broker id:3,host:10.181.18.244,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:48:24 SyncProducer [INFO] Connected to 10.181.18.244:9082 for producing
2015-12-15 12:48:24 SyncProducer [INFO] Disconnecting from 10.181.18.244:9082
2015-12-15 12:48:54 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:49:24 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:49:34 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:49:34 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:49:34 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:49:34 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:49:34 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:50:04 SyncProducer [INFO] Connected to 10.181.18.87:9082 for producing
2015-12-15 12:50:04 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:50:04 ClientUtils$ [WARN] Fetching topic metadata with correlation id 0 for topics [Set(TOPIC_ONE, TOPIC_TWO)] from broker [id:5,host:10.181.18.87,port:9082] failed
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at org.apache.samza.util.ClientUtilTopicMetadataStore.getTopicInfo(ClientUtilTopicMetadataStore.scala:37)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getTopicMetadata(KafkaSystemAdmin.scala:214)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2$$anonfun$6.apply(KafkaSystemAdmin.scala:158)
at org.apache.samza.system.kafka.TopicMetadataCache$.getTopicMetadata(TopicMetadataCache.scala:52)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:155)
at org.apache.samza.system.kafka.KafkaSystemAdmin$$anonfun$getSystemStreamMetadata$2.apply(KafkaSystemAdmin.scala:154)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:153)
at org.apache.samza.system.kafka.KafkaSystemAdmin.getSystemStreamMetadata(KafkaSystemAdmin.scala:142)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:62)
at org.apache.samza.system.StreamMetadataCache$$anonfun$3.apply(StreamMetadataCache.scala:58)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.samza.system.StreamMetadataCache.getStreamMetadata(StreamMetadataCache.scala:58)
at org.apache.samza.container.SamzaContainer$.apply(SamzaContainer.scala:192)
at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
2015-12-15 12:50:04 SyncProducer [INFO] Disconnecting from 10.181.18.87:9082
2015-12-15 12:50:04 ClientUtils$ [INFO] Fetching metadata from broker id:2,host:10.181.18.171,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:50:04 SyncProducer [INFO] Connected to 10.181.18.171:9082 for producing
2015-12-15 12:50:04 SyncProducer [INFO] Disconnecting from 10.181.18.171:9082
2015-12-15 12:50:34 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:51:04 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:51:14 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:51:14 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:51:14 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:51:14 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:51:14 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:10.181.18.223,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:51:14 SyncProducer [INFO] Connected to 10.181.18.223:9082 for producing
2015-12-15 12:51:14 SyncProducer [INFO] Disconnecting from 10.181.18.223:9082
2015-12-15 12:51:44 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:52:14 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:52:24 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:52:24 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:52:24 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:52:24 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:52:24 ClientUtils$ [INFO] Fetching metadata from broker id:1,host:10.181.18.194,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
2015-12-15 12:52:24 SyncProducer [INFO] Connected to 10.181.18.194:9082 for producing
2015-12-15 12:52:24 SyncProducer [INFO] Disconnecting from 10.181.18.194:9082
2015-12-15 12:52:54 SimpleConsumer [INFO] Reconnect due to socket error: java.nio.channels.ClosedChannelException
2015-12-15 12:53:24 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [TOPIC_ONE, TOPIC_TWO] due to java.nio.channels.ClosedChannelException. Retrying.
2015-12-15 12:53:34 VerifiableProperties [INFO] Verifying properties
2015-12-15 12:53:34 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-samza_job_production_1-1-1450180218584-0
2015-12-15 12:53:34 VerifiableProperties [INFO] Property metadata.broker.list is overridden to 10.181.18.223:9082,10.181.18.194:9082,10.181.18.171:9082,10.181.18.244:9082,10.181.18.134:9082,10.181.18.87:9082,10.181.18.193:9082
2015-12-15 12:53:34 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-12-15 12:53:34 ClientUtils$ [INFO] Fetching metadata from broker id:5,host:10.181.18.87,port:9082 with correlation id 0 for 2 topic(s) Set(TOPIC_ONE, TOPIC_TWO)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment