Created
April 30, 2018 21:10
-
-
Save mzeier/8294febd6d113450f3872ba48c6056a8 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[2018-04-30 10:43:14,772: DEBUG/Worker-3] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2018 Pivotal Software, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'direct_reply_to': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@a0d2023c56a4', u'platform': u'Erlang/OTP 20.1.7', u'version': u'3.7.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] | |
[2018-04-30 10:43:14,775: DEBUG/Worker-3] Open OK! | |
[2018-04-30 10:43:14,775: DEBUG/Worker-3] using channel_id: 1 | |
[2018-04-30 10:43:14,777: DEBUG/Worker-3] Channel open | |
[2018-04-30 10:43:14,783: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'69400b2a-bfb0-32ad-b226-6843e3f5da78', u'exchange': u'reply.celery.pidbox'} ticket:39983596-31b9-460b-8858-0eaa30d9521c] | |
[2018-04-30 10:43:14,798: DEBUG/Worker-3] Closed channel #1 | |
2018-04-30 10:43:14,802 INFO awx.main.tasks Workers on tower node 'awx' removed from queues [] and added to queues [] | |
2018-04-30 10:43:14,805 INFO awx.main.tasks Worker on tower node 'awx' updated celery routes {'awx.main.tasks.purge_old_stdout_files': {'queue': 'awx', 'routing_key': 'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': 'awx', 'routing_key': 'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': 'awx', 'routing_key': 'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': 'awx', 'routing_key': 'awx'}} | |
[2018-04-30 10:43:14,812: INFO/MainProcess] Task awx.main.tasks.handle_ha_toplogy_changes[d8eb3773-aa49-4024-b7ca-d9c4770f94a7] succeeded in 0.100395352114s: None | |
[2018-04-30 10:43:19,437: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-04-30 10:43:19,438: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->59b39fb6-3042-49e2-99c2-baa812994741 | |
[2018-04-30 10:43:19,439: DEBUG/Beat] beat: Waking up in 19.99 seconds. | |
[2018-04-30 10:43:19,439: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] expires:[2018-04-30 10:43:39.438352+00:00] | |
[2018-04-30 10:43:19,440: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x188a320> (args:(u'awx.main.scheduler.tasks.run_task_manager', u'59b39fb6-3042-49e2-99c2-baa812994741', [], {}, {u'utc': True, u'is_eager': False, u'chord': None, u'group': None, u'args': [], u'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'awx_private_queue', u'exchange': u'awx_private_queue'}, u'expires': u'2018-04-30T10:43:39.438352+00:00', u'hostname': 'celery@awx', u'task': u'awx.main.scheduler.tasks.run_task_manager', u'callbacks': None, u'correlation_id': u'59b39fb6-3042-49e2-99c2-baa812994741', u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'reply_to': u'b80f3fd4-0b42-3eb5-be2f-4aef597e0d47', u'id': u'59b39fb6-3042-49e2-99c2-baa812994741', u'headers': {}}) kwargs:{}) | |
[2018-04-30 10:43:19,442: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] pid:223 | |
2018-04-30 10:43:19,467 DEBUG awx.main.scheduler Running Tower task manager. | |
2018-04-30 10:43:19,474 DEBUG awx.main.scheduler Starting Scheduler | |
2018-04-30 10:43:19,485 ERROR awx.main.tasks Task awx.main.scheduler.tasks.run_task_manager encountered exception. | |
Traceback (most recent call last): | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task | |
R = retval = fun(*args, **kwargs) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__ | |
return self.run(*args, **kwargs) | |
File "/usr/lib/python2.7/site-packages/awx/main/scheduler/tasks.py", line 31, in run_task_manager | |
TaskManager().schedule() | |
File "/usr/lib/python2.7/site-packages/awx/main/scheduler/task_manager.py", line 641, in schedule | |
wfj.send_notification_templates('succeeded' if wfj.status == 'successful' else 'failed') | |
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ | |
self.gen.throw(type, value, traceback) | |
File "/usr/lib/python2.7/site-packages/awx/main/utils/pglock.py", line 14, in advisory_lock | |
yield internal_lock | |
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ | |
self.gen.throw(type, value, traceback) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django_pglocks/__init__.py", line 80, in advisory_lock | |
cursor.execute(command) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute | |
return self.cursor.execute(sql, params) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__ | |
six.reraise(dj_exc_type, dj_exc_value, traceback) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute | |
return self.cursor.execute(sql) | |
InternalError: current transaction is aborted, commands ignored until end of transaction block | |
[2018-04-30 10:43:19,488: ERROR/MainProcess] Task awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] raised unexpected: InternalError('current transaction is aborted, commands ignored until end of transaction block\n',) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment