celery-connectionreset.txt

[2020-07-31 15:39:10,659: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672//
[2020-07-31 15:39:10,659: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:10,659: DEBUG/MainProcess] | Consumer: Starting Events
[2020-07-31 15:39:10,660: DEBUG/MainProcess] Closed channel #1
[2020-07-31 15:39:10,677: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2020-07-31 15:39:10,680: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:10,680: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-07-31 15:39:10,680: INFO/MainProcess] mingle: searching for neighbors
[2020-07-31 15:39:10,681: DEBUG/MainProcess] using channel_id: 1
[2020-07-31 15:39:10,683: DEBUG/MainProcess] Channel open
[2020-07-31 15:39:10,704: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2020-07-31 15:39:10,706: DEBUG/MainProcess] using channel_id: 1
[2020-07-31 15:39:10,707: DEBUG/MainProcess] Channel open
[2020-07-31 15:39:10,709: DEBUG/MainProcess] Closed channel #1
[2020-07-31 15:39:11,746: INFO/MainProcess] mingle: all alone
[2020-07-31 15:39:11,746: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:11,747: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-07-31 15:39:11,747: DEBUG/MainProcess] using channel_id: 2
[2020-07-31 15:39:11,750: DEBUG/MainProcess] Channel open
[2020-07-31 15:39:11,757: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:11,757: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-07-31 15:39:11,775: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:11,775: DEBUG/MainProcess] | Consumer: Starting Control
[2020-07-31 15:39:11,775: DEBUG/MainProcess] using channel_id: 3
[2020-07-31 15:39:11,777: DEBUG/MainProcess] Channel open
[2020-07-31 15:39:11,785: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:11,785: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-07-31 15:39:11,786: DEBUG/MainProcess] using channel_id: 1
[2020-07-31 15:39:11,787: DEBUG/MainProcess] Channel open
[2020-07-31 15:39:11,788: DEBUG/MainProcess] ^-- substep ok
[2020-07-31 15:39:11,789: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-07-31 15:39:11,789: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2020-07-31 15:39:11,790: DEBUG/MainProcess] basic.qos: prefetch_count->4
[2020-07-31 15:39:11,791: INFO/MainProcess] Received task: long_running_task[623813b0-3b69-4728-a27f-1150a3aa25f1]  
[2020-07-31 15:39:11,794: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fded372ce18> (args:('long_running_task', '623813b0-3b69-4728-a27f-1150a3aa25f1', {'lang': 'py', 'task': 'long_running_task', 'id': '623813b0-3b69-4728-a27f-1150a3aa25f1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '623813b0-3b69-4728-a27f-1150a3aa25f1', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'n': 30}", 'origin': 'gen6@0736f621fc77', 'reply_to': '60b99528-066b-378b-be49-3348e1fda267', 'correlation_id': '623813b0-3b69-4728-a27f-1150a3aa25f1', 'hostname': 'celery@5da7711becb9', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': True}, 'args': [], 'kwargs': {'n': 30}}, '[[], {"n": 30}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-07-31 15:39:11,794: DEBUG/MainProcess] Task accepted: long_running_task[623813b0-3b69-4728-a27f-1150a3aa25f1] pid:6
[2020-07-31 15:39:11,794: WARNING/MainProcess] This task will take: 30 seconds
[2020-07-31 15:39:11,795: WARNING/MainProcess] Task started
[2020-07-31 15:39:11,795: WARNING/MainProcess] 0
[2020-07-31 15:39:21,804: WARNING/MainProcess] 10
[2020-07-31 15:39:31,813: WARNING/MainProcess] 20
[2020-07-31 15:39:41,823: WARNING/MainProcess] > /app/celery_config/celery_tasks.py(122)long_running_task()
-> print("All done")
[2020-07-31 15:39:41,824: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:03,497: WARNING/MainProcess] All done
[2020-07-31 15:40:03,498: WARNING/MainProcess] --Return--
[2020-07-31 15:40:03,498: WARNING/MainProcess] > /app/celery_config/celery_tasks.py(122)long_running_task()->None
-> print("All done")
[2020-07-31 15:40:03,498: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:09,001: WARNING/MainProcess] --Call--
[2020-07-31 15:40:09,005: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(248)__exit__()
-> def __exit__(self, exc_type, exc_value, tb):
[2020-07-31 15:40:09,006: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:17,087: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(249)__exit__()
-> self.pop(exc_value)
[2020-07-31 15:40:17,088: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:18,382: WARNING/MainProcess] --Call--
[2020-07-31 15:40:18,383: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(231)pop()
-> def pop(self, exc=_sentinel):
[2020-07-31 15:40:18,383: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:19,242: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(233)pop()
-> try:
[2020-07-31 15:40:19,242: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:20,680: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(234)pop()
-> self._refcnt -= 1
[2020-07-31 15:40:20,680: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:21,617: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(235)pop()
-> if self._refcnt <= 0:
[2020-07-31 15:40:21,617: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:22,389: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(236)pop()
-> if exc is _sentinel:
[2020-07-31 15:40:22,390: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:23,163: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(238)pop()
-> self.app.do_teardown_appcontext(exc)
[2020-07-31 15:40:23,164: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:40:23,795: WARNING/MainProcess] --Call--
[2020-07-31 15:40:23,798: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/app.py(2302)do_teardown_appcontext()
-> def do_teardown_appcontext(self, exc=_sentinel):
[2020-07-31 15:40:23,798: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:25,664: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/app.py(2317)do_teardown_appcontext()
-> if exc is _sentinel:
[2020-07-31 15:40:25,665: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:26,472: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/app.py(2319)do_teardown_appcontext()
-> for func in reversed(self.teardown_appcontext_funcs):
[2020-07-31 15:40:26,473: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:27,489: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/app.py(2321)do_teardown_appcontext()
-> appcontext_tearing_down.send(self, exc=exc)
[2020-07-31 15:40:27,490: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:30,214: WARNING/MainProcess] --Return--
[2020-07-31 15:40:30,215: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/app.py(2321)do_teardown_appcontext()->None
-> appcontext_tearing_down.send(self, exc=exc)
[2020-07-31 15:40:30,215: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:31,259: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(240)pop()
-> rv = _app_ctx_stack.pop()
[2020-07-31 15:40:31,259: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:32,891: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(241)pop()
-> assert rv is self, "Popped wrong app context.  (%r instead of %r)" % (rv, self)
[2020-07-31 15:40:32,891: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:36,838: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(242)pop()
-> appcontext_popped.send(self.app)
[2020-07-31 15:40:36,839: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:37,835: WARNING/MainProcess] --Return--
[2020-07-31 15:40:37,835: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(242)pop()->None
-> appcontext_popped.send(self.app)
[2020-07-31 15:40:37,835: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:38,913: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(251)__exit__()
-> if BROKEN_PYPY_CTXMGR_EXIT and exc_type is not None:
[2020-07-31 15:40:38,914: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:48,128: WARNING/MainProcess] --Return--
[2020-07-31 15:40:48,129: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/flask/ctx.py(251)__exit__()->None
-> if BROKEN_PYPY_CTXMGR_EXIT and exc_type is not None:
[2020-07-31 15:40:48,129: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:49,491: WARNING/MainProcess] --Return--
[2020-07-31 15:40:49,492: WARNING/MainProcess] > /app/celery_config/celery_worker.py(15)__call__()->None
-> return self.run(*args, **kwargs)
[2020-07-31 15:40:49,492: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:50,545: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(413)trace_task()
-> state = SUCCESS
[2020-07-31 15:40:50,545: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:57,981: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(434)trace_task()
-> try:
[2020-07-31 15:40:57,981: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:40:59,203: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(442)trace_task()
-> callbacks = task.request.callbacks
[2020-07-31 15:40:59,204: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:41:04,458: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(443)trace_task()
-> if callbacks:
[2020-07-31 15:41:04,459: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:41:05,690: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(471)trace_task()
-> chain = task_request.chain
[2020-07-31 15:41:05,690: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:41:07,229: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(472)trace_task()
-> if chain:
[2020-07-31 15:41:07,230: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:41:07,973: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(479)trace_task()
-> mark_as_done(
[2020-07-31 15:41:07,974: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:41:08,625: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(480)trace_task()
-> uuid, retval, task_request, publish_result,
[2020-07-31 15:41:08,626: WARNING/MainProcess] (Pdb)
[2020-07-31 15:41:20,948: DEBUG/Beat] beat: Synchronizing schedule...
[2020-07-31 15:41:20,950: DEBUG/Beat] beat: Waking up in 5.00 minutes.
n
[2020-07-31 15:41:40,365: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2020-07-31 15:41:40,367: DEBUG/MainProcess] using channel_id: 1
[2020-07-31 15:41:40,369: DEBUG/MainProcess] Channel open
[2020-07-31 15:41:40,375: DEBUG/MainProcess] Closed channel #1
[2020-07-31 15:41:40,377: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(485)trace_task()
-> Rstr = saferepr(R, resultrepr_maxsize)
[2020-07-31 15:41:40,377: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:21,324: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(486)trace_task()
-> T = monotonic() - time_start
[2020-07-31 15:42:21,324: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:25,680: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(487)trace_task()
-> if task_on_success:
[2020-07-31 15:42:25,681: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:27,256: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(489)trace_task()
-> if success_receivers:
[2020-07-31 15:42:27,256: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:28,140: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(491)trace_task()
-> if _does_info:
[2020-07-31 15:42:28,140: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:42:32,716: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(492)trace_task()
-> info(LOG_SUCCESS, {
[2020-07-31 15:42:32,717: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:34,465: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(493)trace_task()
-> 'id': uuid,
[2020-07-31 15:42:34,465: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:39,147: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(494)trace_task()
-> 'name': get_task_name(task_request, name),
[2020-07-31 15:42:39,148: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:41,476: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(495)trace_task()
-> 'return_value': Rstr,
[2020-07-31 15:42:41,476: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:42,846: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(496)trace_task()
-> 'runtime': T,
[2020-07-31 15:42:42,847: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:43,783: INFO/MainProcess] Task long_running_task[623813b0-3b69-4728-a27f-1150a3aa25f1] succeeded in 193.88552324401098s: None
[2020-07-31 15:42:43,784: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(500)trace_task()
-> if state not in IGNORE_STATES:
[2020-07-31 15:42:43,784: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:48,776: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(501)trace_task()
-> if task_after_return:
[2020-07-31 15:42:48,776: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:49,662: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(506)trace_task()
-> try:
[2020-07-31 15:42:49,662: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:51,005: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(507)trace_task()
-> if postrun_receivers:
[2020-07-31 15:42:51,006: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:52,627: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(512)trace_task()
-> pop_task()
[2020-07-31 15:42:52,627: WARNING/MainProcess] (Pdb)
s
[2020-07-31 15:42:57,216: WARNING/MainProcess] --Call--
[2020-07-31 15:42:57,220: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(217)pop()
-> def pop(self):
[2020-07-31 15:42:57,220: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:42:58,801: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(223)pop()
-> stack = getattr(self._local, 'stack', None)
[2020-07-31 15:42:58,802: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:03,962: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(224)pop()
-> if stack is None:
[2020-07-31 15:43:03,963: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:04,834: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(226)pop()
-> elif len(stack) == 1:
[2020-07-31 15:43:04,834: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:05,763: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(227)pop()
-> release_local(self._local)
[2020-07-31 15:43:05,764: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:06,659: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(228)pop()
-> return stack[-1]
[2020-07-31 15:43:06,659: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:07,592: WARNING/MainProcess] --Return--
[2020-07-31 15:43:07,592: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/utils/threads.py(228)pop()-><@task: long_...x7fdf9e9e2a20>
-> return stack[-1]
[2020-07-31 15:43:07,593: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:15,539: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(513)trace_task()
-> pop_request()
[2020-07-31 15:43:15,539: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:16,869: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(514)trace_task()
-> if not eager:
[2020-07-31 15:43:16,869: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:18,192: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(515)trace_task()
-> try:
[2020-07-31 15:43:18,192: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:18,944: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(516)trace_task()
-> backend_cleanup()
[2020-07-31 15:43:18,945: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:19,534: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(517)trace_task()
-> loader_cleanup()
[2020-07-31 15:43:19,534: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:20,318: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(532)trace_task()
-> return trace_ok_t(R, I, T, Rstr)
[2020-07-31 15:43:20,319: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:22,297: WARNING/MainProcess] --Return--
[2020-07-31 15:43:22,297: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(532)trace_task()->trace_ok_t(re...retstr='None')
-> return trace_ok_t(R, I, T, Rstr)
[2020-07-31 15:43:22,298: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:23,521: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(614)_fast_trace_task()
-> return (1, R, T) if I else (0, Rstr, T)
[2020-07-31 15:43:23,522: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:28,565: WARNING/MainProcess] --Return--
[2020-07-31 15:43:28,565: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/app/trace.py(614)_fast_trace_task()->(0, 'None', 193.88552324401098)
-> return (1, R, T) if I else (0, Rstr, T)
[2020-07-31 15:43:28,566: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:29,201: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/concurrency/base.py(46)apply_target()
-> callback(ret)
[2020-07-31 15:43:29,201: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:29,932: WARNING/MainProcess] --Return--
[2020-07-31 15:43:29,933: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/concurrency/base.py(46)apply_target()->None
-> callback(ret)
[2020-07-31 15:43:29,933: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:30,593: WARNING/MainProcess] --Return--
[2020-07-31 15:43:30,594: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/concurrency/base.py(158)apply_async()->None
-> **options)
[2020-07-31 15:43:30,594: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:44,480: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/request.py(656)execute_using_pool()
-> self._apply_result = maybe(ref, result)
[2020-07-31 15:43:44,481: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:47,883: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/request.py(657)execute_using_pool()
-> return result
[2020-07-31 15:43:47,884: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:50,144: WARNING/MainProcess] --Return--
[2020-07-31 15:43:50,145: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/request.py(657)execute_using_pool()->None
-> return result
[2020-07-31 15:43:50,145: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:51,169: WARNING/MainProcess] --Return--
[2020-07-31 15:43:51,185: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/worker.py(231)_process_task()->None
-> req.execute_using_pool(self.pool)
[2020-07-31 15:43:51,185: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:55,540: WARNING/MainProcess] --Return--
[2020-07-31 15:43:55,544: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/strategy.py(203)task_message_handler()->None
-> handle(req)
[2020-07-31 15:43:55,544: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:57,583: WARNING/MainProcess] --Return--
[2020-07-31 15:43:57,584: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/consumer/consumer.py(571)on_task_received()->None
-> callbacks,
[2020-07-31 15:43:57,584: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:43:59,807: WARNING/MainProcess] --Return--
[2020-07-31 15:43:59,836: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/messaging.py(624)_receive_callback()->None
-> return on_m(message) if on_m else self.receive(decoded, message)
[2020-07-31 15:43:59,836: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:05,490: WARNING/MainProcess] --Return--
[2020-07-31 15:44:05,491: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/channel.py(1615)_on_basic_deliver()->None
-> fun(msg)
[2020-07-31 15:44:05,491: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:07,455: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(144)dispatch_method()
-> for listener in listeners:
[2020-07-31 15:44:07,455: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:08,583: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(147)dispatch_method()
-> if one_shot:
[2020-07-31 15:44:08,583: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:09,461: WARNING/MainProcess] --Return--
[2020-07-31 15:44:09,461: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(147)dispatch_method()->None
-> if one_shot:
[2020-07-31 15:44:09,462: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:10,150: WARNING/MainProcess] --Return--
[2020-07-31 15:44:10,178: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/connection.py(521)on_inbound_method()->None
-> method_sig, payload, content,
[2020-07-31 15:44:10,178: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:16,267: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/method_framing.py(83)on_frame()
-> return True
[2020-07-31 15:44:16,268: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:17,792: WARNING/MainProcess] --Return--
[2020-07-31 15:44:17,793: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/method_framing.py(83)on_frame()->True
-> return True
[2020-07-31 15:44:17,794: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:18,688: WARNING/MainProcess] --Return--
[2020-07-31 15:44:18,689: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/connection.py(514)blocking_read()->True
-> return self.on_inbound_frame(frame)
[2020-07-31 15:44:18,690: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:20,122: WARNING/MainProcess] --Return--
[2020-07-31 15:44:20,123: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/connection.py(508)drain_events()->None
-> while not self.blocking_read(timeout):
[2020-07-31 15:44:20,123: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:21,179: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(87)wait()
-> while not p.ready:
[2020-07-31 15:44:21,179: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:22,770: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(88)wait()
-> self.connection.drain_events(timeout=timeout)
[2020-07-31 15:44:22,771: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:24,180: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(87)wait()
-> while not p.ready:
[2020-07-31 15:44:24,180: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:25,046: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(90)wait()
-> if p.value:
[2020-07-31 15:44:25,046: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:26,271: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(91)wait()
-> args, kwargs = p.value
[2020-07-31 15:44:26,271: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:27,013: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(92)wait()
-> args = args[1:]  # We are not returning method back
[2020-07-31 15:44:27,014: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:30,022: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(93)wait()
-> return args if returns_tuple else (args and args[0])
[2020-07-31 15:44:30,022: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:30,777: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(95)wait()
-> for i, m in enumerate(method):
[2020-07-31 15:44:30,777: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:31,848: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(96)wait()
-> if prev_p[i] is not None:
[2020-07-31 15:44:31,849: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:32,649: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(99)wait()
-> pending.pop(m, None)
[2020-07-31 15:44:32,649: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:33,251: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(95)wait()
-> for i, m in enumerate(method):
[2020-07-31 15:44:33,251: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:33,841: WARNING/MainProcess] --Return--
[2020-07-31 15:44:33,841: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(95)wait()->()
-> for i, m in enumerate(method):
[2020-07-31 15:44:33,842: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:34,453: WARNING/MainProcess] --Return--
[2020-07-31 15:44:34,454: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py(68)send_method()->()
-> return self.wait(wait, returns_tuple=returns_tuple)
[2020-07-31 15:44:34,455: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:35,133: WARNING/MainProcess] --Return--
[2020-07-31 15:44:35,134: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/amqp/channel.py(1858)basic_qos()->()
-> wait=spec.Basic.QosOk,
[2020-07-31 15:44:35,134: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:35,854: WARNING/MainProcess] --Return--
[2020-07-31 15:44:35,855: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/messaging.py(558)qos()->()
-> apply_global)
[2020-07-31 15:44:35,855: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:36,552: WARNING/MainProcess] --Return--
[2020-07-31 15:44:36,568: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/consumer/tasks.py(47)set_prefetch_count()->()
-> apply_global=qos_global,
[2020-07-31 15:44:36,568: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:37,504: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/common.py(440)set()
-> self.prev = pcount
[2020-07-31 15:44:37,504: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:38,373: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/common.py(441)set()
-> return pcount
[2020-07-31 15:44:38,373: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:39,195: WARNING/MainProcess] --Return--
[2020-07-31 15:44:39,195: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/common.py(441)set()->4
-> return pcount
[2020-07-31 15:44:39,196: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:40,040: WARNING/MainProcess] --Call--
[2020-07-31 15:44:40,041: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/asynchronous/semaphore.py(107)__exit__()
-> def __exit__(self, *exc_info):
[2020-07-31 15:44:40,042: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:41,180: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/asynchronous/semaphore.py(108)__exit__()
-> pass
[2020-07-31 15:44:41,181: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:43,150: WARNING/MainProcess] --Return--
[2020-07-31 15:44:43,150: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/asynchronous/semaphore.py(108)__exit__()->None
-> pass
[2020-07-31 15:44:43,150: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:43,812: WARNING/MainProcess] --Return--
[2020-07-31 15:44:43,813: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/kombu/common.py(446)update()->4
-> return self.set(self.value)
[2020-07-31 15:44:43,813: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:44,456: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/loops.py(82)asynloop()
-> try:
[2020-07-31 15:44:44,456: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:45,064: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/loops.py(83)asynloop()
-> next(loop)
[2020-07-31 15:44:45,065: WARNING/MainProcess] (Pdb)
n
[2020-07-31 15:44:45,734: CRITICAL/MainProcess] Couldn't ack 1, reason:ConnectionResetError(104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/kombu/message.py", line 131, in ack_log_error
    self.ack(multiple=multiple)
  File "/usr/local/lib/python3.6/dist-packages/kombu/message.py", line 126, in ack
    self.channel.basic_ack(self.delivery_tag, multiple=multiple)
  File "/usr/local/lib/python3.6/dist-packages/amqp/channel.py", line 1394, in basic_ack
    spec.Basic.Ack, argsig, (delivery_tag, multiple),
  File "/usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py", line 59, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/usr/local/lib/python3.6/dist-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/usr/local/lib/python3.6/dist-packages/amqp/transport.py", line 305, in write
    self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer
[2020-07-31 15:44:45,754: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2020-07-31 15:44:45,861: DEBUG/MainProcess] heartbeat_tick : for connection 322d70b3846243b3873407dde663574d
[2020-07-31 15:44:45,862: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 3/3, monotonic - 122411.623495975, last_heartbeat_sent - 122411.62349369, heartbeat int. - 60 for connection 322d70b3846243b3873407dde663574d
[2020-07-31 15:44:45,863: WARNING/MainProcess] amqp.exceptions.RecoverableConnectionError: Socket was disconnected
[2020-07-31 15:44:45,863: WARNING/MainProcess] > /usr/local/lib/python3.6/dist-packages/celery/worker/loops.py(83)asynloop()
-> next(loop)
[2020-07-31 15:44:45,864: WARNING/MainProcess] (Pdb)