Skip to content

Instantly share code, notes, and snippets.

@mzeier
Created April 30, 2018 21:10
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mzeier/8294febd6d113450f3872ba48c6056a8 to your computer and use it in GitHub Desktop.
Save mzeier/8294febd6d113450f3872ba48c6056a8 to your computer and use it in GitHub Desktop.
[2018-04-30 10:43:14,772: DEBUG/Worker-3] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2018 Pivotal Software, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'direct_reply_to': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@a0d2023c56a4', u'platform': u'Erlang/OTP 20.1.7', u'version': u'3.7.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US']
[2018-04-30 10:43:14,775: DEBUG/Worker-3] Open OK!
[2018-04-30 10:43:14,775: DEBUG/Worker-3] using channel_id: 1
[2018-04-30 10:43:14,777: DEBUG/Worker-3] Channel open
[2018-04-30 10:43:14,783: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'69400b2a-bfb0-32ad-b226-6843e3f5da78', u'exchange': u'reply.celery.pidbox'} ticket:39983596-31b9-460b-8858-0eaa30d9521c]
[2018-04-30 10:43:14,798: DEBUG/Worker-3] Closed channel #1
2018-04-30 10:43:14,802 INFO awx.main.tasks Workers on tower node 'awx' removed from queues [] and added to queues []
2018-04-30 10:43:14,805 INFO awx.main.tasks Worker on tower node 'awx' updated celery routes {'awx.main.tasks.purge_old_stdout_files': {'queue': 'awx', 'routing_key': 'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': 'awx', 'routing_key': 'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': 'awx', 'routing_key': 'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': 'awx', 'routing_key': 'awx'}}
[2018-04-30 10:43:14,812: INFO/MainProcess] Task awx.main.tasks.handle_ha_toplogy_changes[d8eb3773-aa49-4024-b7ca-d9c4770f94a7] succeeded in 0.100395352114s: None
[2018-04-30 10:43:19,437: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-04-30 10:43:19,438: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->59b39fb6-3042-49e2-99c2-baa812994741
[2018-04-30 10:43:19,439: DEBUG/Beat] beat: Waking up in 19.99 seconds.
[2018-04-30 10:43:19,439: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] expires:[2018-04-30 10:43:39.438352+00:00]
[2018-04-30 10:43:19,440: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x188a320> (args:(u'awx.main.scheduler.tasks.run_task_manager', u'59b39fb6-3042-49e2-99c2-baa812994741', [], {}, {u'utc': True, u'is_eager': False, u'chord': None, u'group': None, u'args': [], u'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'awx_private_queue', u'exchange': u'awx_private_queue'}, u'expires': u'2018-04-30T10:43:39.438352+00:00', u'hostname': 'celery@awx', u'task': u'awx.main.scheduler.tasks.run_task_manager', u'callbacks': None, u'correlation_id': u'59b39fb6-3042-49e2-99c2-baa812994741', u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'reply_to': u'b80f3fd4-0b42-3eb5-be2f-4aef597e0d47', u'id': u'59b39fb6-3042-49e2-99c2-baa812994741', u'headers': {}}) kwargs:{})
[2018-04-30 10:43:19,442: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] pid:223
2018-04-30 10:43:19,467 DEBUG awx.main.scheduler Running Tower task manager.
2018-04-30 10:43:19,474 DEBUG awx.main.scheduler Starting Scheduler
2018-04-30 10:43:19,485 ERROR awx.main.tasks Task awx.main.scheduler.tasks.run_task_manager encountered exception.
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/awx/main/scheduler/tasks.py", line 31, in run_task_manager
TaskManager().schedule()
File "/usr/lib/python2.7/site-packages/awx/main/scheduler/task_manager.py", line 641, in schedule
wfj.send_notification_templates('succeeded' if wfj.status == 'successful' else 'failed')
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/awx/main/utils/pglock.py", line 14, in advisory_lock
yield internal_lock
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django_pglocks/__init__.py", line 80, in advisory_lock
cursor.execute(command)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
InternalError: current transaction is aborted, commands ignored until end of transaction block
[2018-04-30 10:43:19,488: ERROR/MainProcess] Task awx.main.scheduler.tasks.run_task_manager[59b39fb6-3042-49e2-99c2-baa812994741] raised unexpected: InternalError('current transaction is aborted, commands ignored until end of transaction block\n',)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment