pykafka_consumer_traceback

in-slave03.nj.istresearch.com] out: 2015-11-06 11:25:23,434 - pykafka.simpleconsumer - INFO - Autocommitting consumer offset for consumer group scrapy-janitor and topic memex.crawled_firehose
[in-slave02.nj.istresearch.com] out: 2015-11-06 11:25:28,692 - kazoo.recipe.watchers - ERROR - timed out
[in-slave02.nj.istresearch.com] out: Traceback (most recent call last):
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 333, in _get_children
[in-slave02.nj.istresearch.com] out:     result = self._func(children)
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 599, in _brokers_changed
[in-slave02.nj.istresearch.com] out:     self._rebalance()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 454, in _rebalance
[in-slave02.nj.istresearch.com] out:     self.commit_offsets()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 680, in commit_offsets
[in-slave02.nj.istresearch.com] out:     return self._consumer.commit_offsets()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 397, in commit_offsets
[in-slave02.nj.istresearch.com] out:     self._consumer_group, 1, b'pykafka', reqs)
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/broker.py", line 333, in commit_consumer_group_offsets
[in-slave02.nj.istresearch.com] out:     return self._offsets_channel_req_handler.request(req).get(OffsetCommitResponse)
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/handlers.py", line 59, in get
[in-slave02.nj.istresearch.com] out:     raise self.error
[in-slave03.nj.istresearch.com] out: 2015-11-06 11:25:28,695 - kazoo.recipe.watchers - ERROR - timed out
[in-slave02.nj.istresearch.com] out: timeout: timed out
[in-slave02.nj.istresearch.com] out: 2015-11-06 11:25:28,712 - kazoo.handlers.threading - ERROR - Exception in worker queue thread
[in-slave03.nj.istresearch.com] out: Traceback (most recent call last):
[in-slave02.nj.istresearch.com] out: Traceback (most recent call last):
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 333, in _get_children
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 99, in _thread_worker
[in-slave03.nj.istresearch.com] out:     result = self._func(children)
[in-slave02.nj.istresearch.com] out:     func()
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 599, in _brokers_changed
[in-slave03.nj.istresearch.com] out:     self._rebalance()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 196, in <lambda>
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 454, in _rebalance
[in-slave02.nj.istresearch.com] out:     self.callback_queue.put(lambda: callback.func(*callback.args))
[in-slave03.nj.istresearch.com] out:     self.commit_offsets()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 341, in _watcher
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 680, in commit_offsets
[in-slave02.nj.istresearch.com] out:     self._get_children(event)
[in-slave03.nj.istresearch.com] out:     return self._consumer.commit_offsets()
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 36, in wrapper
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 397, in commit_offsets
[in-slave02.nj.istresearch.com] out:     return func(*args, **kwargs)
[in-slave03.nj.istresearch.com] out:     self._consumer_group, 1, b'pykafka', reqs)
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 333, in _get_children
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/broker.py", line 333, in commit_consumer_group_offsets
[in-slave02.nj.istresearch.com] out:     result = self._func(children)
[in-slave03.nj.istresearch.com] out:     return self._offsets_channel_req_handler.request(req).get(OffsetCommitResponse)
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 599, in _brokers_changed
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/handlers.py", line 59, in get
[in-slave02.nj.istresearch.com] out:     self._rebalance()
[in-slave03.nj.istresearch.com] out:     raise self.error
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 454, in _rebalance
[in-slave03.nj.istresearch.com] out: timeout: timed out
[in-slave02.nj.istresearch.com] out:     self.commit_offsets()
[in-slave03.nj.istresearch.com] out: 2015-11-06 11:25:28,710 - kazoo.handlers.threading - ERROR - Exception in worker queue thread
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 680, in commit_offsets
[in-slave03.nj.istresearch.com] out: Traceback (most recent call last):
[in-slave02.nj.istresearch.com] out:     return self._consumer.commit_offsets()
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 99, in _thread_worker
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 397, in commit_offsets
[in-slave03.nj.istresearch.com] out:     func()
[in-slave02.nj.istresearch.com] out:     self._consumer_group, 1, b'pykafka', reqs)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 196, in <lambda>
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/broker.py", line 333, in commit_consumer_group_offsets
[in-slave03.nj.istresearch.com] out:     self.callback_queue.put(lambda: callback.func(*callback.args))
[in-slave02.nj.istresearch.com] out:     return self._offsets_channel_req_handler.request(req).get(OffsetCommitResponse)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 341, in _watcher
[in-slave02.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/handlers.py", line 59, in get
[in-slave03.nj.istresearch.com] out:     self._get_children(event)
[in-slave02.nj.istresearch.com] out:     raise self.error
[in-slave02.nj.istresearch.com] out: timeout: timed out
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 36, in wrapper
[in-slave03.nj.istresearch.com] out:     return func(*args, **kwargs)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/kazoo/recipe/watchers.py", line 333, in _get_children
[in-slave03.nj.istresearch.com] out:     result = self._func(children)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 599, in _brokers_changed
[in-slave03.nj.istresearch.com] out:     self._rebalance()
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 454, in _rebalance
[in-slave03.nj.istresearch.com] out:     self.commit_offsets()
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 680, in commit_offsets
[in-slave03.nj.istresearch.com] out:     return self._consumer.commit_offsets()
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 397, in commit_offsets
[in-slave03.nj.istresearch.com] out:     self._consumer_group, 1, b'pykafka', reqs)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/broker.py", line 333, in commit_consumer_group_offsets
[in-slave03.nj.istresearch.com] out:     return self._offsets_channel_req_handler.request(req).get(OffsetCommitResponse)
[in-slave03.nj.istresearch.com] out:   File "/tmp/virtualenv/virtualenv_root/scrapy-janitor/lib/python2.7/site-packages/pykafka/handlers.py", line 59, in get
[in-slave03.nj.istresearch.com] out:     raise self.error
[in-slave03.nj.istresearch.com] out: timeout: timed out