Created
August 4, 2020 16:26
-
-
Save shenoy-anurag/783e78896e4888c118e40222c63b3354 to your computer and use it in GitHub Desktop.
Celery conn resets once and thus repeats long task once. `BROKER_HEARTBEAT = 600` is the setting that causes this to happen.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2020-08-04 16:14:17,553: INFO/MainProcess] Received task: long_running_task[b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0] | |
| [2020-08-04 16:14:17,554: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f5bf3fe8378> (args:('long_running_task', 'b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0', {'lang': 'py', 'task': 'long_running_task', 'id': 'b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'n': 150}", 'origin': 'gen6@ae9258ba9c7c', 'reply_to': '357fb0b0-5899-39c6-9d44-cf757030071a', 'correlation_id': 'b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0', 'hostname': 'celery@eddee4e4fd64', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}, 'args': [], 'kwargs': {'n': 150}}, '[[], {"n": 150}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{}) | |
| [2020-08-04 16:14:17,554: DEBUG/MainProcess] Task accepted: long_running_task[b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0] pid:6 | |
| [2020-08-04 16:14:17,554: WARNING/MainProcess] This task will take: 150 seconds | |
| [2020-08-04 16:14:17,554: WARNING/MainProcess] Task started | |
| [2020-08-04 16:14:17,554: WARNING/MainProcess] 0 | |
| [2020-08-04 16:14:27,564: WARNING/MainProcess] 10 | |
| [2020-08-04 16:14:37,574: WARNING/MainProcess] 20 | |
| [2020-08-04 16:14:45,627: INFO/Beat] Scheduler: Sending due task update_cache (update_cache) | |
| [2020-08-04 16:14:45,629: DEBUG/Beat] update_cache sent. id->0428fe54-00b9-4b0e-a709-a2013a681aae | |
| [2020-08-04 16:14:45,630: DEBUG/Beat] beat: Waking up in 14.36 seconds. | |
| [2020-08-04 16:14:47,584: WARNING/MainProcess] 30 | |
| [2020-08-04 16:14:57,594: WARNING/MainProcess] 40 | |
| [2020-08-04 16:15:00,004: INFO/Beat] Scheduler: Sending due task celery_config.celery_tasks.logger_upload_file (celery_config.celery_tasks.logger_upload_file) | |
| [2020-08-04 16:15:00,006: DEBUG/Beat] celery_config.celery_tasks.logger_upload_file sent. id->756309f9-ad8b-4240-b3ab-7e950f5bb1be | |
| [2020-08-04 16:15:00,007: DEBUG/Beat] beat: Waking up in 59.98 seconds. | |
| [2020-08-04 16:15:07,603: WARNING/MainProcess] 50 | |
| [2020-08-04 16:15:17,613: WARNING/MainProcess] 60 | |
| [2020-08-04 16:15:27,622: WARNING/MainProcess] 70 | |
| [2020-08-04 16:15:37,631: WARNING/MainProcess] 80 | |
| [2020-08-04 16:15:47,641: WARNING/MainProcess] 90 | |
| [2020-08-04 16:15:57,651: WARNING/MainProcess] 100 | |
| [2020-08-04 16:16:00,048: INFO/Beat] Scheduler: Sending due task celery_config.celery_tasks.logger_upload_file (celery_config.celery_tasks.logger_upload_file) | |
| [2020-08-04 16:16:00,051: DEBUG/Beat] beat: Synchronizing schedule... | |
| [2020-08-04 16:16:00,052: DEBUG/Beat] celery_config.celery_tasks.logger_upload_file sent. id->e98e5423-cdda-4db1-9978-dc573203a649 | |
| [2020-08-04 16:16:00,056: DEBUG/Beat] beat: Waking up in 59.93 seconds. | |
| [2020-08-04 16:16:07,660: WARNING/MainProcess] 110 | |
| [2020-08-04 16:16:17,671: WARNING/MainProcess] 120 | |
| rabbitmq_1 | 2020-08-04 16:16:17.724 [error] <0.2707.0> closing AMQP connection <0.2707.0> (172.19.0.6:35616 -> 172.19.0.4:5672): | |
| rabbitmq_1 | missed heartbeats from client, timeout: 60s | |
| [2020-08-04 16:16:27,680: WARNING/MainProcess] 130 | |
| [2020-08-04 16:16:37,690: WARNING/MainProcess] 140 | |
| [2020-08-04 16:16:47,699: WARNING/MainProcess] All done | |
| [2020-08-04 16:16:47,714: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| [2020-08-04 16:16:47,715: DEBUG/MainProcess] using channel_id: 1 | |
| [2020-08-04 16:16:47,718: DEBUG/MainProcess] Channel open | |
| [2020-08-04 16:16:47,732: DEBUG/MainProcess] Closed channel #1 | |
| [2020-08-04 16:16:47,733: INFO/MainProcess] Task long_running_task[b9a90217-05ed-4eb7-9bcd-d3e3fb9c4ed0] succeeded in 150.17848117099493s: None | |
| [2020-08-04 16:16:47,733: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None] | |
| [2020-08-04 16:16:47,734: CRITICAL/MainProcess] Couldn't ack 10, reason:ConnectionResetError(104, 'Connection reset by peer') | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/message.py", line 131, in ack_log_error | |
| self.ack(multiple=multiple) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/message.py", line 126, in ack | |
| self.channel.basic_ack(self.delivery_tag, multiple=multiple) | |
| File "/usr/local/lib/python3.6/dist-packages/amqp/channel.py", line 1394, in basic_ack | |
| spec.Basic.Ack, argsig, (delivery_tag, multiple), | |
| File "/usr/local/lib/python3.6/dist-packages/amqp/abstract_channel.py", line 59, in send_method | |
| conn.frame_writer(1, self.channel_id, sig, args, content) | |
| File "/usr/local/lib/python3.6/dist-packages/amqp/method_framing.py", line 172, in write_frame | |
| write(view[:offset]) | |
| File "/usr/local/lib/python3.6/dist-packages/amqp/transport.py", line 305, in write | |
| self._write(s) | |
| ConnectionResetError: [Errno 104] Connection reset by peer | |
| [2020-08-04 16:16:47,753: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| [2020-08-04 16:16:47,756: DEBUG/MainProcess] heartbeat_tick : for connection 8e7cb9643d0d41d78806b578c9e954da | |
| [2020-08-04 16:16:47,756: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 3/3, monotonic - 168873.994952409, last_heartbeat_sent - 168873.994950807, heartbeat int. - 60 for connection 8e7cb9643d0d41d78806b578c9e954da | |
| [2020-08-04 16:16:47,756: ERROR/MainProcess] Error cleaning up after event loop: RecoverableConnectionError(None, 'Socket was disconnected', None, '') | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/dist-packages/celery/worker/loops.py", line 83, in asynloop | |
| next(loop) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/asynchronous/hub.py", line 364, in create_loop | |
| cb(*cbargs) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/transport/base.py", line 238, in on_readable | |
| reader(loop) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/transport/base.py", line 218, in _read | |
| raise RecoverableConnectionError('Socket was disconnected') | |
| amqp.exceptions.RecoverableConnectionError: Socket was disconnected | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/dist-packages/celery/worker/loops.py", line 88, in asynloop | |
| hub.reset() | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/asynchronous/hub.py", line 112, in reset | |
| self.close() | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/asynchronous/hub.py", line 265, in close | |
| item() | |
| File "/usr/local/lib/python3.6/dist-packages/vine/promises.py", line 170, in __call__ | |
| return self.throw() | |
| File "/usr/local/lib/python3.6/dist-packages/vine/promises.py", line 167, in __call__ | |
| retval = fun(*final_args, **final_kwargs) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/transport/base.py", line 218, in _read | |
| raise RecoverableConnectionError('Socket was disconnected') | |
| amqp.exceptions.RecoverableConnectionError: Socket was disconnected | |
| [2020-08-04 16:16:47,757: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection... | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/dist-packages/celery/worker/consumer/consumer.py", line 318, in start | |
| blueprint.start(self) | |
| File "/usr/local/lib/python3.6/dist-packages/celery/bootsteps.py", line 119, in start | |
| step.start(parent) | |
| File "/usr/local/lib/python3.6/dist-packages/celery/worker/consumer/consumer.py", line 599, in start | |
| c.loop(*c.loop_args()) | |
| File "/usr/local/lib/python3.6/dist-packages/celery/worker/loops.py", line 83, in asynloop | |
| next(loop) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/asynchronous/hub.py", line 364, in create_loop | |
| cb(*cbargs) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/transport/base.py", line 238, in on_readable | |
| reader(loop) | |
| File "/usr/local/lib/python3.6/dist-packages/kombu/transport/base.py", line 218, in _read | |
| raise RecoverableConnectionError('Socket was disconnected') | |
| amqp.exceptions.RecoverableConnectionError: Socket was disconnected | |
| celery_1 | [2020-08-04 16:09:17,693: DEBUG/MainProcess] | Consumer: Restarting event loop... | |
| celery_1 | [2020-08-04 16:09:17,694: DEBUG/MainProcess] | Consumer: Restarting Gossip... | |
| celery_1 | [2020-08-04 16:09:17,695: DEBUG/MainProcess] | Consumer: Restarting Control... | |
| celery_1 | [2020-08-04 16:09:17,695: DEBUG/MainProcess] | Consumer: Restarting Tasks... | |
| celery_1 | [2020-08-04 16:09:17,695: DEBUG/MainProcess] Canceling task consumer... | |
| celery_1 | [2020-08-04 16:09:17,696: DEBUG/MainProcess] | Consumer: Restarting Mingle... | |
| celery_1 | [2020-08-04 16:09:17,696: DEBUG/MainProcess] | Consumer: Restarting Heart... | |
| celery_1 | [2020-08-04 16:09:17,697: DEBUG/MainProcess] | Consumer: Restarting Events... | |
| celery_1 | [2020-08-04 16:09:17,697: DEBUG/MainProcess] | Consumer: Restarting Connection... | |
| celery_1 | [2020-08-04 16:09:17,698: DEBUG/MainProcess] | Consumer: Starting Connection | |
| rabbitmq_1 | 2020-08-04 16:09:17.714 [info] <0.2707.0> accepting AMQP connection <0.2707.0> (172.19.0.6:35616 -> 172.19.0.4:5672) | |
| celery_1 | [2020-08-04 16:09:17,715: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| rabbitmq_1 | 2020-08-04 16:09:17.718 [info] <0.2707.0> connection <0.2707.0> (172.19.0.6:35616 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/' | |
| celery_1 | [2020-08-04 16:09:17,719: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672// | |
| celery_1 | [2020-08-04 16:09:17,719: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:17,719: DEBUG/MainProcess] | Consumer: Starting Events | |
| celery_1 | [2020-08-04 16:09:17,720: DEBUG/MainProcess] Closed channel #1 | |
| rabbitmq_1 | 2020-08-04 16:09:17.721 [info] <0.2252.0> closing AMQP connection <0.2252.0> (172.19.0.6:35412 -> 172.19.0.4:5672, vhost: '/', user: 'guest') | |
| rabbitmq_1 | 2020-08-04 16:09:17.737 [info] <0.2715.0> accepting AMQP connection <0.2715.0> (172.19.0.6:35618 -> 172.19.0.4:5672) | |
| celery_1 | [2020-08-04 16:09:17,739: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| rabbitmq_1 | 2020-08-04 16:09:17.742 [info] <0.2715.0> connection <0.2715.0> (172.19.0.6:35618 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/' | |
| celery_1 | [2020-08-04 16:09:17,743: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:17,743: DEBUG/MainProcess] | Consumer: Starting Heart | |
| celery_1 | [2020-08-04 16:09:17,744: DEBUG/MainProcess] using channel_id: 1 | |
| celery_1 | [2020-08-04 16:09:17,746: DEBUG/MainProcess] Channel open | |
| celery_1 | [2020-08-04 16:09:17,748: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:17,748: DEBUG/MainProcess] | Consumer: Starting Mingle | |
| celery_1 | [2020-08-04 16:09:17,749: INFO/MainProcess] mingle: searching for neighbors | |
| celery_1 | [2020-08-04 16:09:17,749: DEBUG/MainProcess] using channel_id: 1 | |
| celery_1 | [2020-08-04 16:09:17,751: DEBUG/MainProcess] Channel open | |
| rabbitmq_1 | 2020-08-04 16:09:17.772 [info] <0.2733.0> accepting AMQP connection <0.2733.0> (172.19.0.6:35620 -> 172.19.0.4:5672) | |
| celery_1 | [2020-08-04 16:09:17,774: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| rabbitmq_1 | 2020-08-04 16:09:17.777 [info] <0.2733.0> connection <0.2733.0> (172.19.0.6:35620 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/' | |
| celery_1 | [2020-08-04 16:09:17,778: DEBUG/MainProcess] using channel_id: 1 | |
| celery_1 | [2020-08-04 16:09:17,780: DEBUG/MainProcess] Channel open | |
| celery_1 | [2020-08-04 16:09:17,783: DEBUG/MainProcess] Closed channel #1 | |
| rabbitmq_1 | 2020-08-04 16:09:17.784 [info] <0.2733.0> closing AMQP connection <0.2733.0> (172.19.0.6:35620 -> 172.19.0.4:5672, vhost: '/', user: 'guest') | |
| celery_1 | [2020-08-04 16:09:18,823: INFO/MainProcess] mingle: all alone | |
| celery_1 | [2020-08-04 16:09:18,823: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:18,824: DEBUG/MainProcess] | Consumer: Starting Tasks | |
| celery_1 | [2020-08-04 16:09:18,846: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:18,846: DEBUG/MainProcess] | Consumer: Starting Control | |
| celery_1 | [2020-08-04 16:09:18,846: DEBUG/MainProcess] using channel_id: 2 | |
| celery_1 | [2020-08-04 16:09:18,847: DEBUG/MainProcess] Channel open | |
| celery_1 | [2020-08-04 16:09:18,852: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:18,852: DEBUG/MainProcess] | Consumer: Starting Gossip | |
| celery_1 | [2020-08-04 16:09:18,852: DEBUG/MainProcess] using channel_id: 3 | |
| celery_1 | [2020-08-04 16:09:18,853: DEBUG/MainProcess] Channel open | |
| celery_1 | [2020-08-04 16:09:18,859: DEBUG/MainProcess] ^-- substep ok | |
| celery_1 | [2020-08-04 16:09:18,859: DEBUG/MainProcess] | Consumer: Starting event loop | |
| celery_1 | [2020-08-04 16:09:18,860: DEBUG/MainProcess] | Worker: Hub.register Pool... | |
| celery_1 | [2020-08-04 16:09:18,861: DEBUG/MainProcess] basic.qos: prefetch_count->4 | |
| celery_1 | [2020-08-04 16:09:18,862: INFO/MainProcess] Received task: long_running_task[ab9d3766-6bfa-471b-ac30-4f441765d752] | |
| celery_1 | [2020-08-04 16:09:18,864: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f5bf3fe8378> (args:('long_running_task', 'ab9d3766-6bfa-471b-ac30-4f441765d752', {'lang': 'py', 'task': 'long_running_task', 'id': 'ab9d3766-6bfa-471b-ac30-4f441765d752', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ab9d3766-6bfa-471b-ac30-4f441765d752', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'n': 150}", 'origin': 'gen6@ae9258ba9c7c', 'reply_to': '357fb0b0-5899-39c6-9d44-cf757030071a', 'correlation_id': 'ab9d3766-6bfa-471b-ac30-4f441765d752', 'hostname': 'celery@eddee4e4fd64', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': True}, 'args': [], 'kwargs': {'n': 150}}, '[[], {"n": 150}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{}) | |
| celery_1 | [2020-08-04 16:09:18,865: DEBUG/MainProcess] Task accepted: long_running_task[ab9d3766-6bfa-471b-ac30-4f441765d752] pid:6 | |
| celery_1 | [2020-08-04 16:09:18,865: WARNING/MainProcess] This task will take: 150 seconds | |
| celery_1 | [2020-08-04 16:09:18,865: WARNING/MainProcess] Task started | |
| celery_1 | [2020-08-04 16:09:18,866: WARNING/MainProcess] 0 | |
| celery_1 | [2020-08-04 16:09:28,874: WARNING/MainProcess] 10 | |
| celery_1 | [2020-08-04 16:09:38,883: WARNING/MainProcess] 20 | |
| celery_1 | [2020-08-04 16:09:48,892: WARNING/MainProcess] 30 | |
| celery_1 | [2020-08-04 16:09:58,902: WARNING/MainProcess] 40 | |
| celery_1 | [2020-08-04 16:10:00,047: DEBUG/Beat] beat: Synchronizing schedule... | |
| celery_1 | [2020-08-04 16:10:00,052: INFO/Beat] Scheduler: Sending due task celery_config.celery_tasks.logger_upload_file (celery_config.celery_tasks.logger_upload_file) | |
| celery_1 | [2020-08-04 16:10:00,054: DEBUG/Beat] celery_config.celery_tasks.logger_upload_file sent. id->a7225a22-af6a-49dc-bc5e-c56cd79bf15e | |
| celery_1 | [2020-08-04 16:10:00,054: DEBUG/Beat] beat: Waking up in 59.94 seconds. | |
| celery_1 | [2020-08-04 16:10:08,911: WARNING/MainProcess] 50 | |
| celery_1 | [2020-08-04 16:10:18,920: WARNING/MainProcess] 60 | |
| celery_1 | [2020-08-04 16:10:28,930: WARNING/MainProcess] 70 | |
| celery_1 | [2020-08-04 16:10:38,940: WARNING/MainProcess] 80 | |
| celery_1 | [2020-08-04 16:10:48,951: WARNING/MainProcess] 90 | |
| celery_1 | [2020-08-04 16:10:58,961: WARNING/MainProcess] 100 | |
| celery_1 | [2020-08-04 16:11:00,050: INFO/Beat] Scheduler: Sending due task celery_config.celery_tasks.logger_upload_file (celery_config.celery_tasks.logger_upload_file) | |
| celery_1 | [2020-08-04 16:11:00,054: DEBUG/Beat] celery_config.celery_tasks.logger_upload_file sent. id->878979a8-ba20-40b5-9f82-0f0c66b06d6f | |
| celery_1 | [2020-08-04 16:11:00,055: DEBUG/Beat] beat: Waking up in 59.93 seconds. | |
| celery_1 | [2020-08-04 16:11:08,970: WARNING/MainProcess] 110 | |
| celery_1 | [2020-08-04 16:11:18,979: WARNING/MainProcess] 120 | |
| celery_1 | [2020-08-04 16:11:28,990: WARNING/MainProcess] 130 | |
| celery_1 | [2020-08-04 16:11:38,999: WARNING/MainProcess] 140 | |
| celery_1 | [2020-08-04 16:11:49,007: WARNING/MainProcess] All done | |
| rabbitmq_1 | 2020-08-04 16:11:49.016 [info] <0.2926.0> accepting AMQP connection <0.2926.0> (172.19.0.6:35768 -> 172.19.0.4:5672) | |
| celery_1 | [2020-08-04 16:11:49,017: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@rabbitmq', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.3', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US'] | |
| celery_1 | [2020-08-04 16:11:49,019: DEBUG/MainProcess] using channel_id: 1 | |
| rabbitmq_1 | 2020-08-04 16:11:49.019 [info] <0.2926.0> connection <0.2926.0> (172.19.0.6:35768 -> 172.19.0.4:5672): user 'guest' authenticated and granted access to vhost '/' | |
| celery_1 | [2020-08-04 16:11:49,020: DEBUG/MainProcess] Channel open | |
| celery_1 | [2020-08-04 16:11:49,023: DEBUG/MainProcess] Closed channel #1 | |
| rabbitmq_1 | 2020-08-04 16:11:49.024 [info] <0.2926.0> closing AMQP connection <0.2926.0> (172.19.0.6:35768 -> 172.19.0.4:5672, vhost: '/', user: 'guest') | |
| celery_1 | [2020-08-04 16:11:49,024: INFO/MainProcess] Task long_running_task[ab9d3766-6bfa-471b-ac30-4f441765d752] succeeded in 150.15912106301403s: None |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment