Skip to content

Instantly share code, notes, and snippets.

@psy-q

psy-q/awx_task.log

Last active February 14, 2018 14:41
Show Gist options
  • Save psy-q/f0bb099ecaa8a38a77e268b90d86cbdf to your computer and use it in GitHub Desktop.
Save psy-q/f0bb099ecaa8a38a77e268b90d86cbdf to your computer and use it in GitHub Desktop.
Download ZIP
AWX jobs stay pending forever
Raw
awx_task.log
2018-02-14 14:29:00,211 ERRO pool awx-config-watcher event buffer overflowed, discarding event 43
[2018-02-14 14:29:06,960: INFO/Beat] Scheduler: Sending due task cluster_heartbeat (awx.main.tasks.cluster_node_heartbeat)
[2018-02-14 14:29:06,967: DEBUG/Beat] awx.main.tasks.cluster_node_heartbeat sent. id->4d6324b3-a5f0-4022-827d-4fb098d535ff
[2018-02-14 14:29:06,967: DEBUG/Beat] beat: Waking up now.
[2018-02-14 14:29:07,170: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler)
[2018-02-14 14:29:07,186: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->eaae06c3-5ec9-479e-9640-8e2f26682e69
[2018-02-14 14:29:07,187: DEBUG/Beat] beat: Waking up in 10.00 seconds.
[2018-02-14 14:29:17,201: DEBUG/Beat] beat: Synchronizing schedule...
[2018-02-14 14:29:17,210: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:29:17,218: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->5fa6ebf5-3ea0-4b06-9219-e029f7550c8d
[2018-02-14 14:29:17,220: DEBUG/Beat] beat: Waking up in 19.94 seconds.
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"changed": false,
"elapsed": 0,
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"elapsed": 300,
"msg": "Timeout when waiting for localhost:11211"
}
Using /etc/ansible/ansible.cfg as config file
Raw
awx_web.log
192.168.122.1 - - [14/Feb/2018:14:35:51 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-"
2018/02/14 14:36:08 [error] 25#0: *56 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:08 [warn] 25#0: *56 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:13 [error] 25#0: *56 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:13 [warn] 25#0: *56 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
192.168.122.1 - - [14/Feb/2018:14:36:13 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-"
2018/02/14 14:36:28 [error] 25#0: *59 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:28 [warn] 25#0: *59 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:33 [error] 25#0: *59 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:33 [warn] 25#0: *59 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
192.168.122.1 - - [14/Feb/2018:14:36:33 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-"
2018/02/14 14:36:56 [error] 25#0: *62 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:36:56 [warn] 25#0: *62 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:37:01 [error] 25#0: *62 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
2018/02/14 14:37:01 [warn] 25#0: *62 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local"
192.168.122.1 - - [14/Feb/2018:14:37:01 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-"
Raw
something_waking_up_in_awx_task.log
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
"changed": false,
"elapsed": 0,
"path": null,
"port": 5432,
"search_regex": null,
"state": "started"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"elapsed": 300,
"msg": "Timeout when waiting for localhost:11211"
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
"changed": false,
"elapsed": 300,
"msg": "Timeout when waiting for localhost:5672"
}
Using /etc/ansible/ansible.cfg as config file
[DEPRECATION WARNING]: The sudo command line option has been deprecated in
favor of the "become" command line arguments. This feature will be removed in
version 2.6. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
127.0.0.1 | SUCCESS => {
"changed": false,
"db": "awx"
}
Operations to perform:
Apply all migrations: auth, conf, contenttypes, django_celery_results, main, sessions, sites, social_django, sso, taggit
Running migrations:
No migrations to apply.
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 9, in <module>
load_entry_point('awx==1.0.3.14', 'console_scripts', 'awx-manage')()
File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/shell.py", line 101, in handle
exec(sys.stdin.read())
File "<string>", line 1, in <module>
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 170, in create_superuser
return self._create_user(username, email, password, **extra_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 153, in _create_user
user.save(using=self._db)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 80, in save
super(AbstractBaseUser, self).save(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
force_update=force_update, update_fields=update_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
using=using, raw=raw)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1107, in execute_sql
cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL: Key (username)=(admin) already exists.
Instance already registered awx
Instance Group already registered tower
2018-02-14 14:39:33,079 CRIT Supervisor running as root (no user in config file)
Unlinking stale socket /tmp/supervisor.sock
2018-02-14 14:39:33,407 INFO RPC interface 'supervisor' initialized
2018-02-14 14:39:33,411 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-02-14 14:39:33,412 INFO supervisord started with pid 164
2018-02-14 14:39:34,414 INFO spawned: 'awx-config-watcher' with pid 167
2018-02-14 14:39:34,415 INFO spawned: 'celery' with pid 168
2018-02-14 14:39:34,416 INFO spawned: 'channels-worker' with pid 169
2018-02-14 14:39:34,432 INFO spawned: 'callback-receiver' with pid 170
2018-02-14 14:39:34,573 INFO exited: awx-config-watcher (exit status 0; not expected)
2018-02-14 14:39:35,577 INFO spawned: 'awx-config-watcher' with pid 193
2018-02-14 14:39:35,579 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-02-14 14:39:35,581 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-02-14 14:39:35,581 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-02-14 14:39:35,719 INFO exited: awx-config-watcher (exit status 0; not expected)
2018-02-14 14:39:37,733 INFO spawned: 'awx-config-watcher' with pid 200
2018-02-14 14:39:37,984 INFO exited: awx-config-watcher (exit status 0; not expected)
2018-02-14 14:39:39,306 INFO awx.main.tasks Syncing Schedules
2018-02-14 14:39:39,460 - INFO - runworker - Using single-threaded worker.
2018-02-14 14:39:39,463 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer)
2018-02-14 14:39:39,469 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive
2018-02-14 14:39:39,800 DEBUG awx.main.tasks Registering celery routes for celery@awx
2018-02-14 14:39:39,808 INFO awx.main.tasks Workers on tower node 'awx' added routes {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}}
/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the -u option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
[2018-02-14 14:39:39,832: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2018-02-14 14:39:39,836: DEBUG/MainProcess] | Worker: Building graph...
[2018-02-14 14:39:39,838: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer}
[2018-02-14 14:39:39,852: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2018-02-14 14:39:39,853: DEBUG/MainProcess] | Consumer: Building graph...
[2018-02-14 14:39:39,866: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Gossip, Tasks, Control, Agent, event loop}
2018-02-14 14:39:39,872 WARNING awx.main.tasks Set hostname to celery@awx
[2018-02-14 14:39:39,877: DEBUG/MainProcess] | Worker: Starting Hub
[2018-02-14 14:39:39,877: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,877: DEBUG/MainProcess] | Worker: Starting Pool
[2018-02-14 14:39:39,878: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,878: DEBUG/MainProcess] | Worker: Starting Autoscaler
[2018-02-14 14:39:39,878: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,878: DEBUG/MainProcess] | Worker: Starting Beat
[2018-02-14 14:39:39,880: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,880: DEBUG/MainProcess] | Worker: Starting Consumer
[2018-02-14 14:39:39,881: DEBUG/MainProcess] | Consumer: Starting Connection
[2018-02-14 14:39:39,906: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:39,916: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672/awx
[2018-02-14 14:39:39,916: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,916: DEBUG/MainProcess] | Consumer: Starting Events
[2018-02-14 14:39:39,929: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:39,934: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,936: DEBUG/MainProcess] | Consumer: Starting Heart
[2018-02-14 14:39:39,947: DEBUG/MainProcess] using channel_id: 1
[2018-02-14 14:39:39,956: DEBUG/MainProcess] Channel open
[2018-02-14 14:39:39,959: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:39,960: DEBUG/MainProcess] | Consumer: Starting Mingle
[2018-02-14 14:39:39,960: INFO/MainProcess] mingle: searching for neighbors
[2018-02-14 14:39:39,960: DEBUG/MainProcess] using channel_id: 1
[2018-02-14 14:39:39,965: DEBUG/MainProcess] Channel open
[2018-02-14 14:39:39,993: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:40,002: DEBUG/MainProcess] using channel_id: 1
[2018-02-14 14:39:40,005: DEBUG/MainProcess] Channel open
2018-02-14 14:39:41,010 INFO spawned: 'awx-config-watcher' with pid 206
[2018-02-14 14:39:41,013: INFO/MainProcess] mingle: all alone
[2018-02-14 14:39:41,013: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:41,014: DEBUG/MainProcess] | Consumer: Starting Gossip
[2018-02-14 14:39:41,014: DEBUG/MainProcess] using channel_id: 2
[2018-02-14 14:39:41,014: DEBUG/MainProcess] Channel open
[2018-02-14 14:39:41,022: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:41,022: DEBUG/MainProcess] | Consumer: Starting Tasks
[2018-02-14 14:39:41,064: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:41,064: DEBUG/MainProcess] | Consumer: Starting Control
[2018-02-14 14:39:41,064: DEBUG/MainProcess] using channel_id: 3
[2018-02-14 14:39:41,069: DEBUG/MainProcess] Channel open
[2018-02-14 14:39:41,074: DEBUG/MainProcess] ^-- substep ok
[2018-02-14 14:39:41,074: DEBUG/MainProcess] | Consumer: Starting event loop
[2018-02-14 14:39:41,074: DEBUG/MainProcess] | Worker: Hub.register Autoscaler...
[2018-02-14 14:39:41,075: DEBUG/MainProcess] | Worker: Hub.register Pool...
2018-02-14 14:39:41,078 DEBUG awx.main.tasks Configure celeryd queues task on host celery@awx
[2018-02-14 14:39:41,130: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:41,138: DEBUG/MainProcess] using channel_id: 1
[2018-02-14 14:39:41,141: DEBUG/MainProcess] Channel open
2018-02-14 14:39:41,195 INFO exited: awx-config-watcher (exit status 0; not expected)
[2018-02-14 14:39:41,389: INFO/Beat] beat: Starting...
2018-02-14 14:39:41,392 INFO gave up: awx-config-watcher entered FATAL state, too many start retries too quickly
[2018-02-14 14:39:41,435: DEBUG/Beat] Current schedule:
<ScheduleEntry: purge_stdout_files awx.main.tasks.purge_old_stdout_files() <freq: 7.00 days>
<ScheduleEntry: tower_scheduler awx.main.tasks.awx_periodic_scheduler() <freq: 30.00 seconds>
<ScheduleEntry: authtoken_cleanup awx.main.tasks.cleanup_authtokens() <freq: 30.00 days>
<ScheduleEntry: isolated_heartbeat awx.main.tasks.awx_isolated_heartbeat() <freq: 10.00 minutes>
<ScheduleEntry: task_manager awx.main.scheduler.tasks.run_task_manager() <freq: 20.00 seconds>
<ScheduleEntry: cluster_heartbeat awx.main.tasks.cluster_node_heartbeat() <freq: 1.00 minute>
<ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>
<ScheduleEntry: admin_checks awx.main.tasks.run_administrative_checks() <freq: 30.00 days>
[2018-02-14 14:39:41,439: DEBUG/Beat] beat: Ticking with max interval->1.00 minute
[2018-02-14 14:39:41,452: DEBUG/Beat] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:41,458: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:39:41,469: DEBUG/Beat] using channel_id: 1
[2018-02-14 14:39:41,472: DEBUG/Beat] Channel open
[2018-02-14 14:39:41,476: DEBUG/Beat] beat: Synchronizing schedule...
[2018-02-14 14:39:41,480: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->e733dd87-b9a0-4008-adf6-b79941de4089
[2018-02-14 14:39:41,480: DEBUG/Beat] beat: Waking up in 19.95 seconds.
2018-02-14 14:39:45,214 INFO awx.main.tasks Workers on tower node 'awx' unsubscribed from queues [] and subscribed to queues [u'tower', 'tower_instance_router', u'awx']
[2018-02-14 14:39:45,219: INFO/MainProcess] celery@awx ready.
[2018-02-14 14:39:45,220: DEBUG/MainProcess] basic.qos: prefetch_count->4
[2018-02-14 14:39:45,225: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:0591ceaf-6a23-4930-8284-247c997a1049]
[2018-02-14 14:39:45,236: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'tower', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:32db1876-2615-4dd2-82d8-4953ecf14509]
[2018-02-14 14:39:45,241: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'tower_instance_router', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:2a823f79-8a31-4bcb-be8f-ecb3fc516f4c]
[2018-02-14 14:39:45,243: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'awx', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:e0fc664f-f99e-4f77-8455-7a86e1a8695d]
[2018-02-14 14:39:45,352: INFO/MainProcess] Started consuming from tower
[2018-02-14 14:39:45,363: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c88ef338-7ca3-40a5-adf6-0916f74add94] expires:[2018-02-14 14:07:36.923698+00:00]
[2018-02-14 14:39:45,368: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c88ef338-7ca3-40a5-adf6-0916f74add94]
[2018-02-14 14:39:45,413: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[748a8d64-39e1-4957-8b1e-98e793520a21] expires:[2018-02-14 14:07:56.801677+00:00]
[2018-02-14 14:39:45,417: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[748a8d64-39e1-4957-8b1e-98e793520a21]
[2018-02-14 14:39:45,427: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7dfe6065-c632-464f-83e7-383e87e59ff3] expires:[2018-02-14 14:07:56.925859+00:00]
[2018-02-14 14:39:45,427: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7dfe6065-c632-464f-83e7-383e87e59ff3]
[2018-02-14 14:39:45,432: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a7529e2e-55aa-49ce-8612-118692fc4ad4] expires:[2018-02-14 14:08:16.923693+00:00]
[2018-02-14 14:39:45,433: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a7529e2e-55aa-49ce-8612-118692fc4ad4]
[2018-02-14 14:39:45,440: INFO/MainProcess] Started consuming from tower_instance_router
[2018-02-14 14:39:45,442: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[043418e1-416b-4526-9f59-7a4569c92af7] expires:[2018-02-14 14:08:26.806176+00:00]
[2018-02-14 14:39:45,442: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[043418e1-416b-4526-9f59-7a4569c92af7]
[2018-02-14 14:39:45,448: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b6323ddc-4077-4462-b4f9-58871c63384c] expires:[2018-02-14 14:08:36.936883+00:00]
[2018-02-14 14:39:45,449: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b6323ddc-4077-4462-b4f9-58871c63384c]
[2018-02-14 14:39:45,458: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[0041b78b-462b-4945-b371-d9407467754e] expires:[2018-02-14 14:08:56.815107+00:00]
[2018-02-14 14:39:45,458: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[0041b78b-462b-4945-b371-d9407467754e]
[2018-02-14 14:39:45,462: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[199917dd-a6e2-4df9-93ae-7653a87aae3e] expires:[2018-02-14 14:08:56.944057+00:00]
[2018-02-14 14:39:45,462: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[199917dd-a6e2-4df9-93ae-7653a87aae3e]
[2018-02-14 14:39:45,471: INFO/MainProcess] Started consuming from awx
[2018-02-14 14:39:45,476: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a9d2355f-5623-42b0-9158-38b0c3c74326] expires:[2018-02-14 14:09:16.945055+00:00]
[2018-02-14 14:39:45,478: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a9d2355f-5623-42b0-9158-38b0c3c74326]
[2018-02-14 14:39:45,484: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[19a567b2-e253-41c8-8904-7d9c7d9cabd3] expires:[2018-02-14 14:08:56.661030+00:00]
[2018-02-14 14:39:45,484: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[19a567b2-e253-41c8-8904-7d9c7d9cabd3]
[2018-02-14 14:39:45,493: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ba296241-c082-49f6-985e-a2e9b6d8bf5b] expires:[2018-02-14 14:09:26.814920+00:00]
[2018-02-14 14:39:45,496: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ba296241-c082-49f6-985e-a2e9b6d8bf5b]
[2018-02-14 14:39:45,503: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0f296241-ece6-48f5-aab6-2cf5142c9aa1] expires:[2018-02-14 14:09:36.941452+00:00]
[2018-02-14 14:39:45,504: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0f296241-ece6-48f5-aab6-2cf5142c9aa1]
[2018-02-14 14:39:46,019: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7c72448f-e74a-43ac-b019-4ff22779fd54] expires:[2018-02-14 14:09:56.661826+00:00]
[2018-02-14 14:39:46,022: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7c72448f-e74a-43ac-b019-4ff22779fd54]
[2018-02-14 14:39:46,028: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ed967de9-af53-4a90-b289-706200e2d5e6] expires:[2018-02-14 14:09:56.825003+00:00]
[2018-02-14 14:39:46,031: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ed967de9-af53-4a90-b289-706200e2d5e6]
[2018-02-14 14:39:46,036: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[6f6737f5-7f8c-400e-aa8c-c2b76f51ddfc] expires:[2018-02-14 14:10:56.665331+00:00]
[2018-02-14 14:39:46,039: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[6f6737f5-7f8c-400e-aa8c-c2b76f51ddfc]
[2018-02-14 14:39:46,045: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0ec3ec2e-6cda-4d29-a93e-a213522b4947] expires:[2018-02-14 14:09:56.946058+00:00]
[2018-02-14 14:39:46,047: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0ec3ec2e-6cda-4d29-a93e-a213522b4947]
[2018-02-14 14:39:46,054: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[772a7400-035a-47c2-abc0-8d810e10ae7d] expires:[2018-02-14 14:10:16.956771+00:00]
[2018-02-14 14:39:46,057: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[772a7400-035a-47c2-abc0-8d810e10ae7d]
[2018-02-14 14:39:46,064: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[b5f632e1-7a45-4fed-b56c-7d36036fa93c] expires:[2018-02-14 14:11:56.662976+00:00]
[2018-02-14 14:39:46,064: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[b5f632e1-7a45-4fed-b56c-7d36036fa93c]
[2018-02-14 14:39:46,071: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b99d717e-2fde-4d0e-8d97-1050360208f0] expires:[2018-02-14 14:10:26.821710+00:00]
[2018-02-14 14:39:46,071: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b99d717e-2fde-4d0e-8d97-1050360208f0]
[2018-02-14 14:39:46,079: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[3ce1dcb1-def1-4ba9-af1a-d33e2697de59] expires:[2018-02-14 14:12:56.667788+00:00]
[2018-02-14 14:39:46,079: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[3ce1dcb1-def1-4ba9-af1a-d33e2697de59]
[2018-02-14 14:39:46,084: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e500e0d2-3722-41bc-807b-d01843a40ba0] expires:[2018-02-14 14:10:36.957260+00:00]
[2018-02-14 14:39:46,084: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[e500e0d2-3722-41bc-807b-d01843a40ba0]
[2018-02-14 14:39:46,088: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a74b1767-303e-4e69-afec-f4a6853983e7] expires:[2018-02-14 14:10:56.822199+00:00]
[2018-02-14 14:39:46,089: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a74b1767-303e-4e69-afec-f4a6853983e7]
[2018-02-14 14:39:46,093: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[6c40f218-8c61-4945-ac1d-d9a9aa85bd72] expires:[2018-02-14 14:10:56.955529+00:00]
[2018-02-14 14:39:46,093: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[6c40f218-8c61-4945-ac1d-d9a9aa85bd72]
[2018-02-14 14:39:46,103: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ce1ed84f-ad34-484c-8dc7-56806bb33a91] expires:[2018-02-14 14:11:16.963834+00:00]
[2018-02-14 14:39:46,105: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ce1ed84f-ad34-484c-8dc7-56806bb33a91]
[2018-02-14 14:39:46,112: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[80492b24-211e-459f-8656-22c467c6a1b0] expires:[2018-02-14 14:13:56.846595+00:00]
[2018-02-14 14:39:46,114: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[80492b24-211e-459f-8656-22c467c6a1b0]
[2018-02-14 14:39:46,120: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7310d492-31e6-4b74-a1d8-44fee934be09] expires:[2018-02-14 14:14:56.849140+00:00]
[2018-02-14 14:39:46,122: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7310d492-31e6-4b74-a1d8-44fee934be09]
[2018-02-14 14:39:46,130: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8174788c-dbb5-43b5-97dc-cba1d28c45d8] expires:[2018-02-14 14:15:56.854597+00:00]
[2018-02-14 14:39:46,133: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8174788c-dbb5-43b5-97dc-cba1d28c45d8]
[2018-02-14 14:39:46,140: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[51a232c8-ef59-4bf7-a35f-2dffeaf21c26] expires:[2018-02-14 14:16:56.852644+00:00]
[2018-02-14 14:39:46,140: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[51a232c8-ef59-4bf7-a35f-2dffeaf21c26]
[2018-02-14 14:39:46,149: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[54f44a88-93a6-401b-8a67-1df94bb7bc77] expires:[2018-02-14 14:11:26.826999+00:00]
[2018-02-14 14:39:46,149: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[54f44a88-93a6-401b-8a67-1df94bb7bc77]
[2018-02-14 14:39:46,155: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[94331b5a-747d-413e-9ff0-64ba8899a643] expires:[2018-02-14 14:17:56.854611+00:00]
[2018-02-14 14:39:46,155: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[94331b5a-747d-413e-9ff0-64ba8899a643]
[2018-02-14 14:39:46,161: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c88f36d3-f40d-4bf6-8dd5-eb35c44db829] expires:[2018-02-14 14:11:36.978382+00:00]
[2018-02-14 14:39:46,162: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c88f36d3-f40d-4bf6-8dd5-eb35c44db829]
[2018-02-14 14:39:46,168: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8a87ca2e-7c2e-4bdc-b5d3-4c36141069ff] expires:[2018-02-14 14:18:56.850983+00:00]
[2018-02-14 14:39:46,168: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8a87ca2e-7c2e-4bdc-b5d3-4c36141069ff]
[2018-02-14 14:39:46,172: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7e3784c4-2caf-4d5f-9bab-6d6ff7ad543b] expires:[2018-02-14 14:19:56.853552+00:00]
[2018-02-14 14:39:46,172: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7e3784c4-2caf-4d5f-9bab-6d6ff7ad543b]
[2018-02-14 14:39:46,178: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[1787ded5-bd6f-406e-b71a-8a116d69023d] expires:[2018-02-14 14:11:56.834816+00:00]
[2018-02-14 14:39:46,180: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[1787ded5-bd6f-406e-b71a-8a116d69023d]
[2018-02-14 14:39:46,186: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[05e0edcc-9a78-451f-8e5e-0a36a627d360] expires:[2018-02-14 14:20:56.854025+00:00]
[2018-02-14 14:39:46,188: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[05e0edcc-9a78-451f-8e5e-0a36a627d360]
[2018-02-14 14:39:46,195: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8560b004-f821-4138-9519-49241e6ac0d7] expires:[2018-02-14 14:21:56.858278+00:00]
[2018-02-14 14:39:46,197: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8560b004-f821-4138-9519-49241e6ac0d7]
[2018-02-14 14:39:46,203: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[c7b0250d-e05f-4a4f-9e4a-c50a95bbad1b] expires:[2018-02-14 14:22:56.856823+00:00]
[2018-02-14 14:39:46,203: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[c7b0250d-e05f-4a4f-9e4a-c50a95bbad1b]
[2018-02-14 14:39:46,209: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4f5be35b-b8b7-4982-acfe-520ca6609304] expires:[2018-02-14 14:11:56.980504+00:00]
[2018-02-14 14:39:46,210: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4f5be35b-b8b7-4982-acfe-520ca6609304]
[2018-02-14 14:39:46,217: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4b3bec6f-54b3-4735-be28-70c32d7b9aba] expires:[2018-02-14 14:12:16.975342+00:00]
[2018-02-14 14:39:46,217: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4b3bec6f-54b3-4735-be28-70c32d7b9aba]
[2018-02-14 14:39:46,225: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[bcc9b244-f2bc-4462-9e0d-12938b49c632] expires:[2018-02-14 14:23:56.963322+00:00]
[2018-02-14 14:39:46,225: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[bcc9b244-f2bc-4462-9e0d-12938b49c632]
[2018-02-14 14:39:47,449: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7c5f42b9-0b93-4824-ad04-c27adeed3f6a] expires:[2018-02-14 14:24:56.959598+00:00]
[2018-02-14 14:39:47,451: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7c5f42b9-0b93-4824-ad04-c27adeed3f6a]
[2018-02-14 14:39:47,457: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[23168038-1560-416c-be46-c0df71b238d9] expires:[2018-02-14 14:12:26.835610+00:00]
[2018-02-14 14:39:47,459: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[23168038-1560-416c-be46-c0df71b238d9]
[2018-02-14 14:39:47,466: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[dcb1f885-9a4b-4e14-be00-5a50c60eb4c6] expires:[2018-02-14 14:25:56.964345+00:00]
[2018-02-14 14:39:47,468: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[dcb1f885-9a4b-4e14-be00-5a50c60eb4c6]
[2018-02-14 14:39:47,475: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[37032744-9082-4fd9-9de7-c9e92ede78c0] expires:[2018-02-14 14:12:36.979865+00:00]
[2018-02-14 14:39:47,477: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[37032744-9082-4fd9-9de7-c9e92ede78c0]
[2018-02-14 14:39:47,483: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[a7dcf11c-5956-4072-9da7-2f50bb807c61] expires:[2018-02-14 14:26:56.960077+00:00]
[2018-02-14 14:39:47,486: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[a7dcf11c-5956-4072-9da7-2f50bb807c61]
[2018-02-14 14:39:47,491: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[6837cdc0-0fb0-4e34-9cf6-e786d5ab8cb2] expires:[2018-02-14 14:27:56.966460+00:00]
[2018-02-14 14:39:47,492: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[6837cdc0-0fb0-4e34-9cf6-e786d5ab8cb2]
[2018-02-14 14:39:47,498: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[f1783d08-e598-4849-bed4-0e56fa4747d2] expires:[2018-02-14 14:12:56.840884+00:00]
[2018-02-14 14:39:47,498: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[f1783d08-e598-4849-bed4-0e56fa4747d2]
[2018-02-14 14:39:47,504: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[cea98b40-6df1-48a7-8bd3-0c782516d978] expires:[2018-02-14 14:28:56.966765+00:00]
[2018-02-14 14:39:47,504: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[cea98b40-6df1-48a7-8bd3-0c782516d978]
[2018-02-14 14:39:47,508: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[894fdc2c-584c-44da-b0f7-406e948a57da] expires:[2018-02-14 14:12:56.981899+00:00]
[2018-02-14 14:39:47,509: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[894fdc2c-584c-44da-b0f7-406e948a57da]
[2018-02-14 14:39:47,515: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[4d6324b3-a5f0-4022-827d-4fb098d535ff] expires:[2018-02-14 14:29:56.964320+00:00]
[2018-02-14 14:39:47,516: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[4d6324b3-a5f0-4022-827d-4fb098d535ff]
[2018-02-14 14:39:47,521: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[bd37a68f-9b2a-4153-ad4d-cbefdc982d5c] expires:[2018-02-14 14:13:16.983728+00:00]
[2018-02-14 14:39:47,521: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[bd37a68f-9b2a-4153-ad4d-cbefdc982d5c]
[2018-02-14 14:39:47,526: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[3b63c184-b7f5-4c4c-9864-1596961c8a83] expires:[2018-02-14 14:13:26.843203+00:00]
[2018-02-14 14:39:47,526: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[3b63c184-b7f5-4c4c-9864-1596961c8a83]
[2018-02-14 14:39:47,531: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[d0314510-e072-4e21-a459-78a00c007450] expires:[2018-02-14 14:13:36.982907+00:00]
[2018-02-14 14:39:47,531: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[d0314510-e072-4e21-a459-78a00c007450]
[2018-02-14 14:39:47,539: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9e09f23d-8c88-4b4b-b0b1-34b1bf6db12c] expires:[2018-02-14 14:13:56.845189+00:00]
[2018-02-14 14:39:47,541: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9e09f23d-8c88-4b4b-b0b1-34b1bf6db12c]
[2018-02-14 14:39:47,547: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[723e5bce-47f3-4d15-aca1-eb429793cd0d] expires:[2018-02-14 14:13:56.985640+00:00]
[2018-02-14 14:39:47,550: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[723e5bce-47f3-4d15-aca1-eb429793cd0d]
[2018-02-14 14:39:47,555: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b27fa26e-7638-4468-be8a-2d17625cd8fe] expires:[2018-02-14 14:14:16.992320+00:00]
[2018-02-14 14:39:47,558: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b27fa26e-7638-4468-be8a-2d17625cd8fe]
[2018-02-14 14:39:47,566: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[e896ddab-d18b-48d6-9095-ebef654b5d25] expires:[2018-02-14 14:14:26.856740+00:00]
[2018-02-14 14:39:47,571: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[e896ddab-d18b-48d6-9095-ebef654b5d25]
[2018-02-14 14:39:47,582: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8aa0c58a-8130-46a9-bcbf-3822cf5c40be] expires:[2018-02-14 14:14:36.999752+00:00]
[2018-02-14 14:39:47,583: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8aa0c58a-8130-46a9-bcbf-3822cf5c40be]
[2018-02-14 14:39:47,592: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[131146ff-b714-40ef-9c41-8a40f96d77f6] expires:[2018-02-14 14:14:56.863443+00:00]
[2018-02-14 14:39:47,592: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[131146ff-b714-40ef-9c41-8a40f96d77f6]
[2018-02-14 14:39:47,599: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[58cb6b31-5686-4d2d-85ff-1ed259cc311d] expires:[2018-02-14 14:14:57.006261+00:00]
[2018-02-14 14:39:47,599: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[58cb6b31-5686-4d2d-85ff-1ed259cc311d]
[2018-02-14 14:39:47,605: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4f530856-01ac-4b58-a601-b3f68d2bb316] expires:[2018-02-14 14:15:17.002741+00:00]
[2018-02-14 14:39:47,605: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4f530856-01ac-4b58-a601-b3f68d2bb316]
[2018-02-14 14:39:47,611: INFO/MainProcess] Received task: awx.main.tasks.awx_isolated_heartbeat[98257bcd-61cf-4775-8db7-a83ca2f3152c] expires:[2018-02-14 14:35:06.870074+00:00]
[2018-02-14 14:39:47,611: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_isolated_heartbeat[98257bcd-61cf-4775-8db7-a83ca2f3152c]
[2018-02-14 14:39:47,617: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a5335e06-943a-417f-8261-e4dda17315e4] expires:[2018-02-14 14:15:26.877808+00:00]
[2018-02-14 14:39:47,618: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a5335e06-943a-417f-8261-e4dda17315e4]
[2018-02-14 14:39:47,621: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7aa364a9-9b53-4b0a-8ff8-04c0dd77610a] expires:[2018-02-14 14:15:37.004958+00:00]
[2018-02-14 14:39:47,622: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7aa364a9-9b53-4b0a-8ff8-04c0dd77610a]
[2018-02-14 14:39:47,626: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800]
[2018-02-14 14:39:47,626: INFO/MainProcess] Scaling up 1 processes.
[2018-02-14 14:39:47,734: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_launch', '8951b093-78dd-4743-bbeb-163851db0800', {'origin': 'gen26@awxweb', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_launch', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '8951b093-78dd-4743-bbeb-163851db0800', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(6,)', 'eta': None, 'parent_id': None, u'reply_to': '75961f62-87ce-37a0-8d47-a1c9e3a1eaf2', 'id': '8951b093-78dd-4743-bbeb-163851db0800', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:47,739: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9e6ae3d8-da80-4466-abd8-f359846d3a18] expires:[2018-02-14 14:15:56.893653+00:00]
[2018-02-14 14:39:47,742: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9e6ae3d8-da80-4466-abd8-f359846d3a18]
[2018-02-14 14:39:47,750: DEBUG/ForkPoolWorker-2] Closed channel #1
[2018-02-14 14:39:47,754: DEBUG/ForkPoolWorker-2] Closed channel #1
[2018-02-14 14:39:47,834: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2018-02-14 14:39:47,846: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[139091ef-2e34-40a5-b041-cfb8ab0bb637] expires:[2018-02-14 14:15:57.007459+00:00]
[2018-02-14 14:39:47,849: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[139091ef-2e34-40a5-b041-cfb8ab0bb637]
[2018-02-14 14:39:47,868: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8bb4e51d-8e88-4438-bf75-682ca3505607] expires:[2018-02-14 14:16:17.013451+00:00]
[2018-02-14 14:39:47,869: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8bb4e51d-8e88-4438-bf75-682ca3505607]
[2018-02-14 14:39:47,879: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9feca935-218b-4b31-b4fc-34f19fae17d6] expires:[2018-02-14 14:16:26.890092+00:00]
[2018-02-14 14:39:47,880: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9feca935-218b-4b31-b4fc-34f19fae17d6]
[2018-02-14 14:39:47,886: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fd9d5959-c3ef-4fc0-b74a-92fd0554d2a1] expires:[2018-02-14 14:16:37.017863+00:00]
[2018-02-14 14:39:47,886: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fd9d5959-c3ef-4fc0-b74a-92fd0554d2a1]
[2018-02-14 14:39:47,892: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[6385cee9-fe34-4fb0-bb73-8635c211c143] expires:[2018-02-14 14:16:56.903719+00:00]
[2018-02-14 14:39:47,892: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[6385cee9-fe34-4fb0-bb73-8635c211c143]
[2018-02-14 14:39:47,902: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[acdf26ce-a740-432a-aefb-607708b5c2a3] expires:[2018-02-14 14:16:57.020229+00:00]
[2018-02-14 14:39:47,902: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[acdf26ce-a740-432a-aefb-607708b5c2a3]
2018-02-14 14:39:47,915 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:39:47,919: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[61e9eb72-863c-4ede-b9cf-d0390f74f015] expires:[2018-02-14 14:17:17.032842+00:00]
2018-02-14 14:39:47,922 DEBUG awx.main.scheduler Failing inconsistent running jobs.
[2018-02-14 14:39:47,922: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[61e9eb72-863c-4ede-b9cf-d0390f74f015]
[2018-02-14 14:39:47,935: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a046f09b-5965-4ccf-97f4-01458f5bf123] expires:[2018-02-14 14:17:26.899325+00:00]
[2018-02-14 14:39:47,938: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a046f09b-5965-4ccf-97f4-01458f5bf123]
[2018-02-14 14:39:47,944: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8b23cb55-28a8-4bcb-abe0-a3e3957b30c2] expires:[2018-02-14 14:17:37.038289+00:00]
[2018-02-14 14:39:47,947: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8b23cb55-28a8-4bcb-abe0-a3e3957b30c2]
[2018-02-14 14:39:47,967: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800] pid:207
[2018-02-14 14:39:47,968: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[91866090-16fe-4a61-8541-ca63aa41eedf] expires:[2018-02-14 14:17:56.913881+00:00]
[2018-02-14 14:39:47,968: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[91866090-16fe-4a61-8541-ca63aa41eedf]
[2018-02-14 14:39:47,982: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[38f9a9fc-562a-4d90-a8e3-6aa6793d42a4] expires:[2018-02-14 14:17:57.042199+00:00]
[2018-02-14 14:39:47,985: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[38f9a9fc-562a-4d90-a8e3-6aa6793d42a4]
[2018-02-14 14:39:47,989: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:47,992: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2b18e67e-417f-4070-a417-ed6e6aca51be] expires:[2018-02-14 14:18:17.043270+00:00]
[2018-02-14 14:39:47,995: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2b18e67e-417f-4070-a417-ed6e6aca51be]
[2018-02-14 14:39:47,999: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2018-02-14 14:39:48,008: DEBUG/ForkPoolWorker-2] Channel open
[2018-02-14 14:39:48,030: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:48,033: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2018-02-14 14:39:48,037: DEBUG/ForkPoolWorker-2] Channel open
[2018-02-14 14:39:48,050: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[5b8d1963-e5e9-4ab4-aae7-6bf590d04654] expires:[2018-02-14 14:18:26.913365+00:00]
[2018-02-14 14:39:48,050: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[5b8d1963-e5e9-4ab4-aae7-6bf590d04654]
[2018-02-14 14:39:48,059: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[73505add-ba4e-41e1-ae56-f0183a114ea9] expires:[2018-02-14 14:18:37.041372+00:00]
[2018-02-14 14:39:48,059: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[73505add-ba4e-41e1-ae56-f0183a114ea9]
[2018-02-14 14:39:48,063: DEBUG/MainProcess] pidbox received method active() [reply_to:{u'routing_key': u'a5d25d51-18e0-3571-bb05-aa52c89277ee', u'exchange': u'reply.celery.pidbox'} ticket:6465f4b0-51ec-4c58-b827-a7904a0e3651]
[2018-02-14 14:39:48,063: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer')
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message
self.node.handle_message(body, message)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message
return self.dispatch(**body)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch
ticket=ticket)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply
serializer=self.mailbox.serializer)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply
**opts
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 1734, in _basic_publish
(0, exchange, routing_key, mandatory, immediate), msg
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write
self._write(s)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 104] Connection reset by peer
[2018-02-14 14:39:48,067: DEBUG/MainProcess] Closed channel #3
[2018-02-14 14:39:48,067: DEBUG/MainProcess] using channel_id: 3
[2018-02-14 14:39:48,068: DEBUG/MainProcess] Channel open
[2018-02-14 14:39:48,072: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[3f6e3d80-0922-4782-b0f6-ac35ae08f2c5] expires:[2018-02-14 14:18:56.924039+00:00]
[2018-02-14 14:39:48,073: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[3f6e3d80-0922-4782-b0f6-ac35ae08f2c5]
[2018-02-14 14:39:48,083: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2f2e1c91-bb61-4644-a205-7b1614d53e43] expires:[2018-02-14 14:18:57.045288+00:00]
[2018-02-14 14:39:48,085: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2f2e1c91-bb61-4644-a205-7b1614d53e43]
[2018-02-14 14:39:48,091: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0b835b02-8553-47b9-8c80-d2079a1f666c] expires:[2018-02-14 14:19:17.042000+00:00]
[2018-02-14 14:39:48,091: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0b835b02-8553-47b9-8c80-d2079a1f666c]
[2018-02-14 14:39:48,098: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b0a5cee7-2f6b-485b-972b-8a44d4c94687] expires:[2018-02-14 14:19:26.921786+00:00]
[2018-02-14 14:39:48,099: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b0a5cee7-2f6b-485b-972b-8a44d4c94687]
[2018-02-14 14:39:48,105: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[79d2db55-03cd-431f-a2ec-a0769006a02a] expires:[2018-02-14 14:19:37.040522+00:00]
[2018-02-14 14:39:48,107: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[79d2db55-03cd-431f-a2ec-a0769006a02a]
[2018-02-14 14:39:48,113: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[0642cc88-2d4b-44c1-bbbd-ad3b4a50ab7a] expires:[2018-02-14 14:19:56.926535+00:00]
[2018-02-14 14:39:48,114: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[0642cc88-2d4b-44c1-bbbd-ad3b4a50ab7a]
[2018-02-14 14:39:48,121: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fba37cec-3389-4253-b979-26327ff28d40] expires:[2018-02-14 14:19:57.043013+00:00]
[2018-02-14 14:39:48,121: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fba37cec-3389-4253-b979-26327ff28d40]
[2018-02-14 14:39:48,126: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[aa5b8b07-7754-4021-9ecf-1ac9230e3328] expires:[2018-02-14 14:20:17.054016+00:00]
[2018-02-14 14:39:48,127: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[aa5b8b07-7754-4021-9ecf-1ac9230e3328]
[2018-02-14 14:39:48,134: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[2f5ced45-c722-410e-84ef-f71efa60fce8] expires:[2018-02-14 14:20:26.923924+00:00]
[2018-02-14 14:39:48,135: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[2f5ced45-c722-410e-84ef-f71efa60fce8]
[2018-02-14 14:39:48,139: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[48621839-d6f0-483d-b4a4-df5c320a5bf5] expires:[2018-02-14 14:20:37.058344+00:00]
[2018-02-14 14:39:48,139: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[48621839-d6f0-483d-b4a4-df5c320a5bf5]
[2018-02-14 14:39:48,144: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[2796819f-e082-4e02-80b8-1dcd57048ea7] expires:[2018-02-14 14:20:56.937793+00:00]
[2018-02-14 14:39:48,144: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[2796819f-e082-4e02-80b8-1dcd57048ea7]
[2018-02-14 14:39:48,149: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4a00ef89-15e3-4d63-b17c-00e4fc903c46] expires:[2018-02-14 14:20:57.064591+00:00]
[2018-02-14 14:39:48,149: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4a00ef89-15e3-4d63-b17c-00e4fc903c46]
[2018-02-14 14:39:48,153: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b0ff083a-3867-443e-8b4d-d59c0a312f54] expires:[2018-02-14 14:21:17.065031+00:00]
[2018-02-14 14:39:48,154: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b0ff083a-3867-443e-8b4d-d59c0a312f54]
[2018-02-14 14:39:48,157: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a77a9add-d0b3-41e4-bae6-c2cb07a64e75] expires:[2018-02-14 14:21:26.934691+00:00]
[2018-02-14 14:39:48,158: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a77a9add-d0b3-41e4-bae6-c2cb07a64e75]
[2018-02-14 14:39:48,164: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c84b0b87-ff9b-4534-a33c-4a830a5af044] expires:[2018-02-14 14:21:37.060661+00:00]
[2018-02-14 14:39:48,167: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c84b0b87-ff9b-4534-a33c-4a830a5af044]
[2018-02-14 14:39:48,175: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[6a0dbc1e-1b72-4fc1-a342-1dad67131d5c] expires:[2018-02-14 14:21:56.945838+00:00]
[2018-02-14 14:39:48,180: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[6a0dbc1e-1b72-4fc1-a342-1dad67131d5c]
[2018-02-14 14:39:48,190: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2f8d0048-7fd0-4a34-bfa5-410aa8ed16f2] expires:[2018-02-14 14:21:57.067142+00:00]
[2018-02-14 14:39:48,190: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2f8d0048-7fd0-4a34-bfa5-410aa8ed16f2]
[2018-02-14 14:39:48,199: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8a683299-debd-4fee-96fc-22bdbcd15d9f] expires:[2018-02-14 14:22:17.072385+00:00]
[2018-02-14 14:39:48,199: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8a683299-debd-4fee-96fc-22bdbcd15d9f]
[2018-02-14 14:39:48,207: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[731aa059-b04a-429a-837e-99a42650e474] expires:[2018-02-14 14:22:26.942150+00:00]
[2018-02-14 14:39:48,207: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[731aa059-b04a-429a-837e-99a42650e474]
[2018-02-14 14:39:48,213: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[9ad2e84f-543c-4476-baae-4e1f36546a3b] expires:[2018-02-14 14:22:37.079048+00:00]
[2018-02-14 14:39:48,214: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[9ad2e84f-543c-4476-baae-4e1f36546a3b]
[2018-02-14 14:39:48,218: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[e11c265c-d145-492e-97a4-c71b4617a340] expires:[2018-02-14 14:22:56.945959+00:00]
[2018-02-14 14:39:48,218: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[e11c265c-d145-492e-97a4-c71b4617a340]
[2018-02-14 14:39:48,222: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fedc34b3-9d6f-4688-93b8-c0ae3b47d7e4] expires:[2018-02-14 14:22:57.076440+00:00]
[2018-02-14 14:39:48,223: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fedc34b3-9d6f-4688-93b8-c0ae3b47d7e4]
[2018-02-14 14:39:48,230: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f61599a0-ff55-4c57-920f-e3aecb0a48e3] expires:[2018-02-14 14:23:17.090884+00:00]
[2018-02-14 14:39:48,231: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f61599a0-ff55-4c57-920f-e3aecb0a48e3]
[2018-02-14 14:39:48,239: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[1049bd07-4015-41c7-8130-1412d7a2783d] expires:[2018-02-14 14:23:26.947478+00:00]
[2018-02-14 14:39:48,239: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[1049bd07-4015-41c7-8130-1412d7a2783d]
[2018-02-14 14:39:48,245: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a69ab1af-1fd3-4b51-9fe8-b1f67fc9f598] expires:[2018-02-14 14:23:37.104587+00:00]
[2018-02-14 14:39:48,248: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a69ab1af-1fd3-4b51-9fe8-b1f67fc9f598]
[2018-02-14 14:39:48,254: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[cc3d218d-ddd8-42ad-bfb6-50ffd675637b] expires:[2018-02-14 14:23:56.945156+00:00]
[2018-02-14 14:39:48,254: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[cc3d218d-ddd8-42ad-bfb6-50ffd675637b]
[2018-02-14 14:39:48,260: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[40442b38-c59b-44d3-99bb-64e0abd72f07] expires:[2018-02-14 14:23:57.106369+00:00]
[2018-02-14 14:39:48,263: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[40442b38-c59b-44d3-99bb-64e0abd72f07]
[2018-02-14 14:39:48,268: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f7d29095-7dfc-4a87-9eb7-5b127498bf6b] expires:[2018-02-14 14:24:17.116393+00:00]
[2018-02-14 14:39:48,271: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f7d29095-7dfc-4a87-9eb7-5b127498bf6b]
[2018-02-14 14:39:48,284: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[90e29f26-9d4b-4170-9f66-828e0ca4294a] expires:[2018-02-14 14:24:26.946368+00:00]
[2018-02-14 14:39:48,284: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[90e29f26-9d4b-4170-9f66-828e0ca4294a]
[2018-02-14 14:39:48,290: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f93a2799-a1ce-4852-84b6-908a1084b767] expires:[2018-02-14 14:24:37.119256+00:00]
[2018-02-14 14:39:48,290: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f93a2799-a1ce-4852-84b6-908a1084b767]
[2018-02-14 14:39:48,295: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[970a32e8-d01c-4c1e-a241-42645a6ae86e] expires:[2018-02-14 14:24:56.947182+00:00]
[2018-02-14 14:39:48,295: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[970a32e8-d01c-4c1e-a241-42645a6ae86e]
[2018-02-14 14:39:48,299: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0b2a7d19-37dc-4882-b7a5-c74723261cc8] expires:[2018-02-14 14:24:57.119025+00:00]
[2018-02-14 14:39:48,300: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0b2a7d19-37dc-4882-b7a5-c74723261cc8]
[2018-02-14 14:39:48,303: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[af5630bc-039f-40df-9653-de0a4c4ded13] expires:[2018-02-14 14:25:17.123895+00:00]
[2018-02-14 14:39:48,304: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[af5630bc-039f-40df-9653-de0a4c4ded13]
[2018-02-14 14:39:48,310: INFO/MainProcess] Received task: awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] expires:[2018-02-14 14:45:06.870985+00:00]
[2018-02-14 14:39:48,312: INFO/MainProcess] Scaling up 1 processes.
[2018-02-14 14:39:48,414: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_isolated_heartbeat', 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_isolated_heartbeat', 'group': None, 'root_id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:45:06.870985+00:00', u'correlation_id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': 'd4c178d7-71ce-36c2-a67c-abc95292f38d', 'id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:48,419: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[4e56d0ca-6990-44bc-8592-6c73d8440a58] expires:[2018-02-14 14:25:26.980125+00:00]
[2018-02-14 14:39:48,421: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[4e56d0ca-6990-44bc-8592-6c73d8440a58]
[2018-02-14 14:39:48,521: DEBUG/MainProcess] basic.qos: prefetch_count->12
[2018-02-14 14:39:48,529: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7faf048b-ecce-4e78-bba1-719ebca166df] expires:[2018-02-14 14:25:37.127133+00:00]
[2018-02-14 14:39:48,535: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7faf048b-ecce-4e78-bba1-719ebca166df]
[2018-02-14 14:39:48,556: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[74bcb58b-9d46-4633-867e-3894b3ca8ae6] expires:[2018-02-14 14:25:56.991142+00:00]
[2018-02-14 14:39:48,556: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[74bcb58b-9d46-4633-867e-3894b3ca8ae6]
[2018-02-14 14:39:48,562: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a67ac9e1-a9f0-4aab-af49-096fee1ab3d7] expires:[2018-02-14 14:25:57.128793+00:00]
[2018-02-14 14:39:48,563: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a67ac9e1-a9f0-4aab-af49-096fee1ab3d7]
[2018-02-14 14:39:48,568: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ab1cf795-3def-4b60-a5de-09ecefd1e2ea] expires:[2018-02-14 14:26:17.137292+00:00]
[2018-02-14 14:39:48,568: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ab1cf795-3def-4b60-a5de-09ecefd1e2ea]
[2018-02-14 14:39:48,577: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[d7b9d8ce-a39c-4809-a041-c4f58b1efb50] expires:[2018-02-14 14:26:26.990658+00:00]
[2018-02-14 14:39:48,578: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[d7b9d8ce-a39c-4809-a041-c4f58b1efb50]
[2018-02-14 14:39:48,584: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c693eedc-0daa-4da6-9575-ec7de6a1dba4] expires:[2018-02-14 14:26:37.136958+00:00]
[2018-02-14 14:39:48,584: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c693eedc-0daa-4da6-9575-ec7de6a1dba4]
[2018-02-14 14:39:48,591: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[69128d40-77e3-480c-a4ce-f4c40a98e699] expires:[2018-02-14 14:26:57.001989+00:00]
[2018-02-14 14:39:48,591: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[69128d40-77e3-480c-a4ce-f4c40a98e699]
[2018-02-14 14:39:48,598: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[806b8cce-d9d8-486d-8bc2-ff48a512f092] expires:[2018-02-14 14:26:57.142839+00:00]
[2018-02-14 14:39:48,598: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[806b8cce-d9d8-486d-8bc2-ff48a512f092]
[2018-02-14 14:39:48,603: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e453574b-ecf2-46f8-b631-0feacd52628c] expires:[2018-02-14 14:27:17.153026+00:00]
[2018-02-14 14:39:48,603: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[e453574b-ecf2-46f8-b631-0feacd52628c]
[2018-02-14 14:39:48,609: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ec75781c-cb03-4d28-9770-37d431102560] expires:[2018-02-14 14:27:26.997756+00:00]
[2018-02-14 14:39:48,609: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ec75781c-cb03-4d28-9770-37d431102560]
[2018-02-14 14:39:48,618: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125]
[2018-02-14 14:39:48,619: INFO/MainProcess] Scaling up 1 processes.
2018-02-14 14:39:48,619 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks.
[2018-02-14 14:39:48,645: INFO/ForkPoolWorker-3] Task awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] succeeded in 0.0323591410006s: None
[2018-02-14 14:39:48,763: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_launch', '323f9e43-4f90-4efe-bec0-a46c60326125', {'origin': 'gen26@awxweb', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_launch', 'group': None, 'root_id': '323f9e43-4f90-4efe-bec0-a46c60326125', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '323f9e43-4f90-4efe-bec0-a46c60326125', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(7,)', 'eta': None, 'parent_id': None, u'reply_to': '75961f62-87ce-37a0-8d47-a1c9e3a1eaf2', 'id': '323f9e43-4f90-4efe-bec0-a46c60326125', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:48,771: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ee1642c3-c826-47ee-a1bd-7ac7558919cd] expires:[2018-02-14 14:27:37.162120+00:00]
[2018-02-14 14:39:48,771: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ee1642c3-c826-47ee-a1bd-7ac7558919cd]
[2018-02-14 14:39:48,943: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a3d03326-ca82-41d4-ab26-d9cff55bab44] expires:[2018-02-14 14:27:57.151092+00:00]
[2018-02-14 14:39:48,950: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a3d03326-ca82-41d4-ab26-d9cff55bab44]
[2018-02-14 14:39:48,977: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[45b096be-28ff-4395-8c40-027b55d4a0d3] expires:[2018-02-14 14:27:57.159716+00:00]
[2018-02-14 14:39:48,977: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[45b096be-28ff-4395-8c40-027b55d4a0d3]
[2018-02-14 14:39:48,985: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[6a97de86-4e3b-4f7d-aa97-c3eda0e423d7] expires:[2018-02-14 14:28:17.160192+00:00]
[2018-02-14 14:39:48,985: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[6a97de86-4e3b-4f7d-aa97-c3eda0e423d7]
[2018-02-14 14:39:48,993: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[4d2069fc-f816-4088-af28-a37bf2dbac00] expires:[2018-02-14 14:28:27.159909+00:00]
[2018-02-14 14:39:48,995: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[4d2069fc-f816-4088-af28-a37bf2dbac00]
[2018-02-14 14:39:49,000: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] pid:208
[2018-02-14 14:39:49,001: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2018-02-14 14:39:49,004: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[d4efc552-6f69-4123-8ffe-97c7fe098370] expires:[2018-02-14 14:28:37.169903+00:00]
[2018-02-14 14:39:49,005: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[d4efc552-6f69-4123-8ffe-97c7fe098370]
[2018-02-14 14:39:49,012: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b83ed07d-5c7f-4f56-9269-753bb886b6a6] expires:[2018-02-14 14:28:57.176527+00:00]
[2018-02-14 14:39:49,012: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b83ed07d-5c7f-4f56-9269-753bb886b6a6]
[2018-02-14 14:39:49,017: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8bf5744c-d993-4ce6-94cc-d8675996c20b] expires:[2018-02-14 14:28:57.194499+00:00]
[2018-02-14 14:39:49,017: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8bf5744c-d993-4ce6-94cc-d8675996c20b]
[2018-02-14 14:39:49,023: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[cd9c89c9-7d8e-4ea2-ab24-78e8180ab63c] expires:[2018-02-14 14:29:17.205705+00:00]
[2018-02-14 14:39:49,023: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[cd9c89c9-7d8e-4ea2-ab24-78e8180ab63c]
[2018-02-14 14:39:49,028: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[eaae06c3-5ec9-479e-9640-8e2f26682e69] expires:[2018-02-14 14:29:27.178238+00:00]
[2018-02-14 14:39:49,028: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[eaae06c3-5ec9-479e-9640-8e2f26682e69]
[2018-02-14 14:39:49,035: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[5fa6ebf5-3ea0-4b06-9219-e029f7550c8d] expires:[2018-02-14 14:29:37.212391+00:00]
[2018-02-14 14:39:49,035: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[5fa6ebf5-3ea0-4b06-9219-e029f7550c8d]
[2018-02-14 14:39:49,043: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] expires:[2018-02-14 14:40:01.459842+00:00]
2018-02-14 14:39:49,046 ERROR awx.main.scheduler Failed to retrieve active tasks from celery
[2018-02-14 14:39:49,052: INFO/MainProcess] Scaling up 1 processes.
2018-02-14 14:39:49,166 DEBUG awx.main.scheduler Starting project_update 6 (pending) in group tower (remaining_capacity=4)
2018-02-14 14:39:49,197 INFO awx.main.scheduler Submitting project_update 6 (waiting) to instance group 1.
2018-02-14 14:39:49,245 DEBUG awx.main.scheduler project_update 6 (waiting) consumed 1 capacity units from tower with prior total of 0
2018-02-14 14:39:49,253 DEBUG awx.main.scheduler project_update 7 (pending) is blocked from running
[2018-02-14 14:39:49,262: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'e733dd87-b9a0-4008-adf6-b79941de4089', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'e733dd87-b9a0-4008-adf6-b79941de4089', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:01.459842+00:00', u'correlation_id': 'e733dd87-b9a0-4008-adf6-b79941de4089', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'e733dd87-b9a0-4008-adf6-b79941de4089', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:49,290: DEBUG/MainProcess] basic.qos: prefetch_count->20
[2018-02-14 14:39:49,301: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] pid:208
[2018-02-14 14:39:49,302: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125] pid:209
2018-02-14 14:39:49,359 DEBUG awx.main.scheduler Running Tower task manager.
2018-02-14 14:39:49,398 DEBUG awx.main.scheduler Starting Scheduler
2018-02-14 14:39:49,403 DEBUG awx.main.scheduler Not running scheduler, another task holds lock
[2018-02-14 14:39:49,414: INFO/ForkPoolWorker-4] Task awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125] succeeded in 0.14597921s: None
[2018-02-14 14:39:49,430: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:49,434: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2018-02-14 14:39:49,438: DEBUG/ForkPoolWorker-2] Channel open
[2018-02-14 14:39:49,455: INFO/MainProcess] Received task: awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31]
[2018-02-14 14:39:49,455: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.run_project_update', '415acd10-4f91-43f0-8ef1-c9686b2fbc31', {'origin': 'gen207@awx', 'lang': 'py', 'task': 'awx.main.tasks.run_project_update', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '[6]', 'eta': None, 'parent_id': '8951b093-78dd-4743-bbeb-163851db0800', u'reply_to': '76df055f-4dd9-3f0d-9c89-9acd797180be', 'id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_success", "subtask_type": null, "kwargs": {"task_actual": {"type": "project_update", "id": 6}}, "args": [], "options": {}, "immutable": false}], "errbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_error", "subtask_type": null, "kwargs": {"subtasks":... kwargs:{})
[2018-02-14 14:39:49,463: DEBUG/MainProcess] Task accepted: awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31] pid:210
[2018-02-14 14:39:49,487: INFO/ForkPoolWorker-2] Task awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800] succeeded in 1.590700005s: None
2018-02-14 14:39:49,529 DEBUG awx.main.scheduler project_update 7 (pending) is blocked from running
[2018-02-14 14:39:49,558: INFO/ForkPoolWorker-3] Task awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] succeeded in 0.287292411001s: None
[2018-02-14 14:39:49,581: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:49,584: DEBUG/ForkPoolWorker-5] using channel_id: 1
[2018-02-14 14:39:49,588: DEBUG/ForkPoolWorker-5] Channel open
[2018-02-14 14:39:49,596: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:49,599: DEBUG/ForkPoolWorker-5] using channel_id: 1
[2018-02-14 14:39:49,603: DEBUG/ForkPoolWorker-5] Channel open
[2018-02-14 14:39:51,142: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:51,148: DEBUG/ForkPoolWorker-5] using channel_id: 1
[2018-02-14 14:39:51,157: DEBUG/ForkPoolWorker-5] Channel open
2018-02-14 14:39:55,915 INFO awx.main.tasks project_update 6 (running) finished running, producing 40 events.
[2018-02-14 14:39:56,095: INFO/MainProcess] Received task: awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888]
[2018-02-14 14:39:56,098: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_work_success', '8072962d-7fbe-45b4-beca-e9664a46b888', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_work_success', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '8072962d-7fbe-45b4-beca-e9664a46b888', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(None,)', 'eta': None, 'parent_id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': '8072962d-7fbe-45b4-beca-e9664a46b888', 'kwargsrepr': "{'task_actual': {'type': 'project_update', 'id': 6}}"}, u'[[null], {"task_actual": {"type": "project_update", "id": 6}}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:56,114: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888] pid:208
[2018-02-14 14:39:56,147: INFO/ForkPoolWorker-5] Task awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31] succeeded in 6.686883101s: None
[2018-02-14 14:39:56,219: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:56,221: DEBUG/ForkPoolWorker-3] using channel_id: 1
[2018-02-14 14:39:56,226: DEBUG/ForkPoolWorker-3] Channel open
[2018-02-14 14:39:56,231: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e]
[2018-02-14 14:39:56,231: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_complete', 'da44f23a-f322-4f21-860e-a2ec0f8c843e', {'origin': 'gen208@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_complete', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(6,)', 'eta': None, 'parent_id': '8072962d-7fbe-45b4-beca-e9664a46b888', u'reply_to': 'ce922c0b-ed65-3190-8808-a32fdcd4136b', 'id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:39:56,234: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e] pid:210
[2018-02-14 14:39:56,249: INFO/ForkPoolWorker-3] Task awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888] succeeded in 0.136103733s: None
2018-02-14 14:39:56,273 DEBUG awx.main.scheduler Starting Scheduler
2018-02-14 14:39:56,352 DEBUG awx.main.scheduler Starting project_update 7 (pending) in group tower (remaining_capacity=4)
2018-02-14 14:39:56,361 INFO awx.main.scheduler Submitting project_update 7 (waiting) to instance group 1.
2018-02-14 14:39:56,425 DEBUG awx.main.scheduler project_update 7 (waiting) consumed 1 capacity units from tower with prior total of 0
[2018-02-14 14:39:56,461: INFO/MainProcess] Received task: awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4]
[2018-02-14 14:39:56,461: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.run_project_update', '18324028-a169-4f04-a943-c5b331544df4', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.run_project_update', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '18324028-a169-4f04-a943-c5b331544df4', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '[7]', 'eta': None, 'parent_id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': '18324028-a169-4f04-a943-c5b331544df4', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_success", "subtask_type": null, "kwargs": {"task_actual": {"type": "project_update", "id": 7}}, "args": [], "options": {}, "immutable": false}], "errbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_error", "subtask_type": null, "kwargs": {"subtasks":... kwargs:{})
[2018-02-14 14:39:56,473: DEBUG/MainProcess] Task accepted: awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4] pid:208
[2018-02-14 14:39:56,501: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e] succeeded in 0.269563346s: None
[2018-02-14 14:39:56,642: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:56,647: DEBUG/ForkPoolWorker-3] using channel_id: 1
[2018-02-14 14:39:56,652: DEBUG/ForkPoolWorker-3] Channel open
[2018-02-14 14:39:58,499: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:39:58,506: DEBUG/ForkPoolWorker-3] using channel_id: 1
[2018-02-14 14:39:58,511: DEBUG/ForkPoolWorker-3] Channel open
[2018-02-14 14:40:01,075: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:01,075: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 192/515, monotonic - 5233.07401304, last_heartbeat_sent - 5233.07400563, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:01,485: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:40:01,547: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] expires:[2018-02-14 14:40:21.529910+00:00]
[2018-02-14 14:40:01,551: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '540dd27b-eb95-442a-9a2b-b8edaa23ddac', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:21.529910+00:00', u'correlation_id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:01,551: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->540dd27b-eb95-442a-9a2b-b8edaa23ddac
[2018-02-14 14:40:01,560: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler)
[2018-02-14 14:40:01,564: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] pid:210
2018-02-14 14:40:01,572 DEBUG awx.main.scheduler Running Tower task manager.
[2018-02-14 14:40:01,571: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->ef10b988-3826-448a-a2d1-82d787821018
[2018-02-14 14:40:01,578: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] expires:[2018-02-14 14:40:21.568358+00:00]
[2018-02-14 14:40:01,583: DEBUG/Beat] beat: Waking up in 19.88 seconds.
[2018-02-14 14:40:01,586: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', 'ef10b988-3826-448a-a2d1-82d787821018', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': 'ef10b988-3826-448a-a2d1-82d787821018', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:21.568358+00:00', u'correlation_id': 'ef10b988-3826-448a-a2d1-82d787821018', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'ef10b988-3826-448a-a2d1-82d787821018', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
2018-02-14 14:40:01,599 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:40:01,626: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] pid:207
[2018-02-14 14:40:01,666: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] succeeded in 0.111874957s: None
2018-02-14 14:40:01,726 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:07:06.819513+00:00
2018-02-14 14:40:01,755 INFO awx.main.tasks project_update 7 (running) finished running, producing 40 events.
[2018-02-14 14:40:01,799: INFO/ForkPoolWorker-2] Task awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] succeeded in 0.160589495s: None
[2018-02-14 14:40:01,901: INFO/MainProcess] Received task: awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7]
[2018-02-14 14:40:01,902: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_work_success', '09395a96-635d-4f4f-aa1f-1e8209828be7', {'origin': 'gen208@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_work_success', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '09395a96-635d-4f4f-aa1f-1e8209828be7', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(None,)', 'eta': None, 'parent_id': '18324028-a169-4f04-a943-c5b331544df4', u'reply_to': 'ce922c0b-ed65-3190-8808-a32fdcd4136b', 'id': '09395a96-635d-4f4f-aa1f-1e8209828be7', 'kwargsrepr': "{'task_actual': {'type': 'project_update', 'id': 7}}"}, u'[[null], {"task_actual": {"type": "project_update", "id": 7}}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:01,907: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7] pid:207
[2018-02-14 14:40:01,936: INFO/ForkPoolWorker-3] Task awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4] succeeded in 5.46970923s: None
[2018-02-14 14:40:02,063: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977]
[2018-02-14 14:40:02,064: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_complete', '6ad5014e-869d-400a-9e38-6de724254977', {'origin': 'gen207@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_complete', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '6ad5014e-869d-400a-9e38-6de724254977', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(7,)', 'eta': None, 'parent_id': '09395a96-635d-4f4f-aa1f-1e8209828be7', u'reply_to': '3cdbea91-bc1e-3caf-b23a-f02313f7d41c', 'id': '6ad5014e-869d-400a-9e38-6de724254977', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:02,071: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977] pid:209
[2018-02-14 14:40:02,093: INFO/ForkPoolWorker-2] Task awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7] succeeded in 0.181390193s: None
2018-02-14 14:40:02,125 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:40:02,175: INFO/ForkPoolWorker-4] Task awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977] succeeded in 0.0991270140003s: None
[2018-02-14 14:40:21,075: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:21,080: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 192/515, now - 196/557, monotonic - 5253.07912095, last_heartbeat_sent - 5253.07909177, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:21,490: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:40:21,499: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->bf9f9083-6225-4ef1-971c-bf89289870f6
[2018-02-14 14:40:21,502: DEBUG/Beat] beat: Waking up in 10.05 seconds.
[2018-02-14 14:40:21,502: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] expires:[2018-02-14 14:40:41.492106+00:00]
[2018-02-14 14:40:21,503: INFO/MainProcess] Scaling down 3 processes.
[2018-02-14 14:40:21,506: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'bf9f9083-6225-4ef1-971c-bf89289870f6', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:41.492106+00:00', u'correlation_id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:21,506: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2018-02-14 14:40:21,597: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] pid:210
2018-02-14 14:40:21,620 DEBUG awx.main.scheduler Running Tower task manager.
2018-02-14 14:40:21,631 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:40:21,653: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] succeeded in 0.0604734569997s: None
[2018-02-14 14:40:31,560: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler)
[2018-02-14 14:40:31,571: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] expires:[2018-02-14 14:40:51.566258+00:00]
[2018-02-14 14:40:31,576: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', '8ec215d7-a4e7-4938-8968-52a1fcc492f2', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:51.566258+00:00', u'correlation_id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:31,589: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] pid:210
[2018-02-14 14:40:31,593: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->8ec215d7-a4e7-4938-8968-52a1fcc492f2
[2018-02-14 14:40:31,599: INFO/Beat] Scheduler: Sending due task cluster_heartbeat (awx.main.tasks.cluster_node_heartbeat)
2018-02-14 14:40:31,606 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:40:01.716576+00:00
[2018-02-14 14:40:31,617: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] expires:[2018-02-14 14:41:21.605482+00:00]
[2018-02-14 14:40:31,618: INFO/MainProcess] Scaling up 1 processes.
[2018-02-14 14:40:31,616: DEBUG/Beat] awx.main.tasks.cluster_node_heartbeat sent. id->13957fad-e323-47fa-993a-2313a6463bd0
[2018-02-14 14:40:31,623: DEBUG/Beat] beat: Waking up in 9.86 seconds.
[2018-02-14 14:40:31,641: INFO/ForkPoolWorker-5] Task awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] succeeded in 0.0645465070002s: None
[2018-02-14 14:40:31,788: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.cluster_node_heartbeat', '13957fad-e323-47fa-993a-2313a6463bd0', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.cluster_node_heartbeat', 'group': None, 'root_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'awx', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.605482+00:00', u'correlation_id': '13957fad-e323-47fa-993a-2313a6463bd0', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '13957fad-e323-47fa-993a-2313a6463bd0', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:31,792: DEBUG/MainProcess] basic.qos: prefetch_count->12
[2018-02-14 14:40:31,803: DEBUG/MainProcess] Task accepted: awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] pid:210
2018-02-14 14:40:31,806 DEBUG awx.main.tasks Cluster node heartbeat task.
[2018-02-14 14:40:31,820: INFO/MainProcess] Received task: awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce]
[2018-02-14 14:40:31,820: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_ha_toplogy_changes', 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_ha_toplogy_changes', 'group': None, 'root_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u'tower_broadcast_all'}, 'expires': None, u'correlation_id': 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:31,829: INFO/ForkPoolWorker-5] Task awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] succeeded in 0.031104312s: None
[2018-02-14 14:40:31,833: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce] pid:210
2018-02-14 14:40:31,852 DEBUG awx.main.tasks Reconfigure celeryd queues task on host celery@awx
[2018-02-14 14:40:31,881: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:40:31,885: DEBUG/ForkPoolWorker-5] using channel_id: 1
[2018-02-14 14:40:31,888: DEBUG/ForkPoolWorker-5] Channel open
[2018-02-14 14:40:31,892: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'90aadbb1-f68f-3e58-9267-63708d9df664', u'exchange': u'reply.celery.pidbox'} ticket:fdfaca9a-ec0f-4259-be61-0ae8b2dacac5]
[2018-02-14 14:40:31,906: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US']
[2018-02-14 14:40:31,909: DEBUG/MainProcess] using channel_id: 1
[2018-02-14 14:40:31,913: DEBUG/MainProcess] Channel open
2018-02-14 14:40:31,921 INFO awx.main.tasks Workers on tower node 'awx' removed from queues [] and added to queues []
2018-02-14 14:40:31,925 INFO awx.main.tasks Worker on tower node 'awx' updated celery routes {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}}
[2018-02-14 14:40:31,930: INFO/ForkPoolWorker-5] Task awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce] succeeded in 0.0928196149998s: None
[2018-02-14 14:40:41,081: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:41,086: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 196/557, now - 202/604, monotonic - 5273.08450469, last_heartbeat_sent - 5273.08447044, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:40:41,490: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:40:41,495: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] expires:[2018-02-14 14:41:01.493104+00:00]
[2018-02-14 14:40:41,496: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '088a9957-5a4c-44e7-a343-c7a857934b53', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '088a9957-5a4c-44e7-a343-c7a857934b53', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:01.493104+00:00', u'correlation_id': '088a9957-5a4c-44e7-a343-c7a857934b53', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '088a9957-5a4c-44e7-a343-c7a857934b53', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:40:41,499: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->088a9957-5a4c-44e7-a343-c7a857934b53
2018-02-14 14:40:41,510 DEBUG awx.main.scheduler Running Tower task manager.
[2018-02-14 14:40:41,513: DEBUG/Beat] beat: Waking up in 20.04 seconds.
[2018-02-14 14:40:41,514: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] pid:443
2018-02-14 14:40:41,524 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:40:41,551: INFO/ForkPoolWorker-6] Task awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] succeeded in 0.0511622899994s: None
[2018-02-14 14:41:01,086: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:41:01,091: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 202/604, now - 203/637, monotonic - 5293.09034378, last_heartbeat_sent - 5293.09031439, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:41:01,574: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler)
[2018-02-14 14:41:01,592: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->d11a467b-be71-4c17-b682-2969859b4dc1
[2018-02-14 14:41:01,599: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] expires:[2018-02-14 14:41:21.580943+00:00]
[2018-02-14 14:41:01,600: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:41:01,601: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', 'd11a467b-be71-4c17-b682-2969859b4dc1', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': 'd11a467b-be71-4c17-b682-2969859b4dc1', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.580943+00:00', u'correlation_id': 'd11a467b-be71-4c17-b682-2969859b4dc1', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'd11a467b-be71-4c17-b682-2969859b4dc1', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:41:01,617: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] pid:443
2018-02-14 14:41:01,623 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:40:31.601264+00:00
[2018-02-14 14:41:01,626: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] expires:[2018-02-14 14:41:21.606234+00:00]
[2018-02-14 14:41:01,627: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.606234+00:00', u'correlation_id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:41:01,626: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865
[2018-02-14 14:41:01,627: DEBUG/Beat] beat: Waking up in 19.96 seconds.
[2018-02-14 14:41:01,632: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] pid:210
2018-02-14 14:41:01,633 DEBUG awx.main.scheduler Running Tower task manager.
2018-02-14 14:41:01,638 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:41:01,651: INFO/ForkPoolWorker-6] Task awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] succeeded in 0.045172829s: None
[2018-02-14 14:41:01,665: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] succeeded in 0.037361273s: None
[2018-02-14 14:41:11,076: INFO/MainProcess] Scaling down 2 processes.
[2018-02-14 14:41:21,092: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:41:21,098: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 203/637, now - 205/673, monotonic - 5313.09638642, last_heartbeat_sent - 5313.09635675, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774
[2018-02-14 14:41:21,610: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-14 14:41:21,627: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->4668edd8-b5b6-4aec-af2a-6f458760ec45
[2018-02-14 14:41:21,625: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] expires:[2018-02-14 14:41:41.615373+00:00]
[2018-02-14 14:41:21,634: DEBUG/Beat] beat: Waking up in 9.95 seconds.
[2018-02-14 14:41:21,635: INFO/MainProcess] Scaling up 1 processes.
[2018-02-14 14:41:21,713: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '4668edd8-b5b6-4aec-af2a-6f458760ec45', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:41.615373+00:00', u'correlation_id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2018-02-14 14:41:21,717: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2018-02-14 14:41:21,724: DEBUG/ForkPoolWorker-7] Closed channel #1
[2018-02-14 14:41:21,827: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] pid:444
2018-02-14 14:41:21,836 DEBUG awx.main.scheduler Running Tower task manager.
2018-02-14 14:41:21,842 DEBUG awx.main.scheduler Starting Scheduler
[2018-02-14 14:41:21,877: INFO/ForkPoolWorker-7] Task awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] succeeded in 0.0467407699998s: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment