Created
January 25, 2014 04:11
-
-
Save mattwilliamson/8611765 to your computer and use it in GitHub Desktop.
Celery Greenlet worker death.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2014-01-24 22:17:36,558: INFO/MainProcess] Task myservice.email.tasks.poll_email_status[cf72ef6d-effa-449e-b866-ee1c3707ce25] succeeded in 2.47090099938s: SM778890846d03dd72f29749f6a7cf517e | |
| [2014-01-24 22:17:36,874: WARNING/MainProcess] Cannot process event on channel 'celeryev': {'pattern': None, 'type': 'message', 'channel': 'celeryev', 'data': '{"body": "eyJ1dGNvZmZzZXQiOiAwLCAidHlwZSI6ICJ0YXNrLXN0YXJ0ZWQiLCAidXVpZCI6ICIyOWVhODQx\nMC02ODk5LTQ1ZTEtYTg3NC1iM2ExOTE1MzRmODgiLCAiY2xvY2siOiA4MTU5NzYsID4+PiBlc3Rh\nbXAiOiAxMzkwNjAwNTE5LjQ5NTMzMywgImhvc3RuYW1lIjogImdyZWVubGV0c0BpbmZsaWdodDMu\nbXlob3N0Lm5ldCIsICJwaWQiOiAzOTYxfQ==", "headers": {"hostname": "greenlets@inflight3.myservice.net"}, "content-type": "application/json", "properties": {"body_encoding": "base64", "delivery_info": {"priority": 0, "routing_key": "task.started", "exchange": "celeryev"}, "delivery_mode'} | |
| Traceback (most recent call last): | |
| File "/www/api.myservice.com/lib/python2.6/site-packages/kombu/transport/redis.py", line 493, in _receive | |
| message = loads(bytes_to_str(payload['data'])) | |
| File "/www/api.myservice.com/lib/python2.6/site-packages/anyjson/__init__.py", line 135, in loads | |
| return implementation.loads(value) | |
| File "/www/api.myservice.com/lib/python2.6/site-packages/anyjson/__init__.py", line 99, in loads | |
| return self._decode(s) | |
| File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads | |
| return _default_decoder.decode(s) | |
| File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode | |
| obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | |
| File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode | |
| obj, end = self._scanner.iterscan(s, **kw).next() | |
| File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan | |
| rval, next_pos = action(m, context) | |
| File "/usr/lib64/python2.6/json/decoder.py", line 183, in JSONObject | |
| value, end = iterscan(s, idx=end, context=context).next() | |
| File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan | |
| rval, next_pos = action(m, context) | |
| File "/usr/lib64/python2.6/json/decoder.py", line 177, in JSONObject | |
| key, end = scanstring(s, end, encoding, strict) | |
| ValueError: Unterminated string starting at: line 1 column 519 (char 519) | |
| [2014-01-24 22:17:37,944: DEBUG/MainProcess] | Worker: Closing Pool... | |
| [2014-01-24 22:17:37,944: DEBUG/MainProcess] | Worker: Closing Consumer... | |
| [2014-01-24 22:17:37,944: DEBUG/MainProcess] | Worker: Stopping Consumer... | |
| [2014-01-24 22:17:37,945: DEBUG/MainProcess] | Consumer: Closing Connection... | |
| [2014-01-24 22:17:37,945: DEBUG/MainProcess] | Consumer: Closing Events... | |
| [2014-01-24 22:17:37,945: DEBUG/MainProcess] | Consumer: Closing Mingle... | |
| [2014-01-24 22:17:37,945: DEBUG/MainProcess] | Consumer: Closing Gossip... | |
| [2014-01-24 22:17:37,946: DEBUG/MainProcess] | Consumer: Closing Control... | |
| [2014-01-24 22:17:37,946: DEBUG/MainProcess] | Consumer: Closing Heart... | |
| [2014-01-24 22:17:37,946: DEBUG/MainProcess] | Consumer: Closing Tasks... | |
| [2014-01-24 22:17:37,946: DEBUG/MainProcess] | Consumer: Closing event loop... | |
| [2014-01-24 22:17:37,946: DEBUG/MainProcess] | Consumer: Stopping event loop... | |
| [2014-01-24 22:17:37,947: DEBUG/MainProcess] | Consumer: Stopping Tasks... | |
| [2014-01-24 22:17:37,947: DEBUG/MainProcess] Cancelling task consumer... | |
| [2014-01-24 22:17:37,947: DEBUG/MainProcess] | Consumer: Stopping Heart... | |
| [2014-01-24 22:17:37,948: DEBUG/MainProcess] | Consumer: Stopping Control... | |
| [2014-01-24 22:17:37,948: DEBUG/MainProcess] Waiting for broadcast thread to shutdown... | |
| [2014-01-24 22:17:37,948: DEBUG/MainProcess] [increment_volume_detection] (account_sid: TrapCall) checking if high volume reached | |
| [2014-01-24 22:17:37,957: INFO/MainProcess] Task myservice.tess_api.tasks.do_callback_task[e0f708cf-2f55-4cea-96f6-f08f018e496d] succeeded in 2.2534910012s: <Response></Response> | |
| [2014-01-24 22:17:37,958: DEBUG/MainProcess] [increment_volume_detection] (cache_count: 1) | |
| [2014-01-24 22:17:37,959: DEBUG/MainProcess] | Consumer: Stopping Gossip... | |
| [2014-01-24 22:17:37,961: DEBUG/MainProcess] | Consumer: Stopping Mingle... | |
| [2014-01-24 22:17:37,961: DEBUG/MainProcess] | Consumer: Stopping Events... | |
| [2014-01-24 22:17:37,961: DEBUG/MainProcess] | Consumer: Stopping Connection... | |
| [2014-01-24 22:17:37,961: DEBUG/MainProcess] | Worker: Stopping Pool... | |
| [2014-01-24 22:17:37,979: INFO/MainProcess] Task myservice.email.tasks.poll_email_status[88a5f8d2-bd03-410a-8112-c8e9ede43f6c] succeeded in 3.50435500033s: SM77889084424cc2f3ea1a42f688aad38b | |
| [2014-01-24 22:17:37,979: DEBUG/MainProcess] | Consumer: Shutdown Tasks... | |
| [2014-01-24 22:17:37,979: DEBUG/MainProcess] Cancelling task consumer... | |
| [2014-01-24 22:17:37,979: DEBUG/MainProcess] Closing consumer channel... | |
| [2014-01-24 22:17:37,980: DEBUG/MainProcess] | Consumer: Shutdown Heart... | |
| [2014-01-24 22:17:37,980: DEBUG/MainProcess] | Consumer: Shutdown Control... | |
| [2014-01-24 22:17:37,980: DEBUG/MainProcess] | Consumer: Shutdown Gossip... | |
| [2014-01-24 22:17:37,980: DEBUG/MainProcess] | Consumer: Shutdown Events... | |
| [2014-01-24 22:17:37,981: DEBUG/MainProcess] | Consumer: Shutdown Connection... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm having the exact same problem. Any insights to share?