Last active
February 14, 2018 14:41
-
-
Save psy-q/f0bb099ecaa8a38a77e268b90d86cbdf to your computer and use it in GitHub Desktop.
AWX jobs stay pending forever
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2018-02-14 14:29:00,211 ERRO pool awx-config-watcher event buffer overflowed, discarding event 43 | |
[2018-02-14 14:29:06,960: INFO/Beat] Scheduler: Sending due task cluster_heartbeat (awx.main.tasks.cluster_node_heartbeat) | |
[2018-02-14 14:29:06,967: DEBUG/Beat] awx.main.tasks.cluster_node_heartbeat sent. id->4d6324b3-a5f0-4022-827d-4fb098d535ff | |
[2018-02-14 14:29:06,967: DEBUG/Beat] beat: Waking up now. | |
[2018-02-14 14:29:07,170: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler) | |
[2018-02-14 14:29:07,186: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->eaae06c3-5ec9-479e-9640-8e2f26682e69 | |
[2018-02-14 14:29:07,187: DEBUG/Beat] beat: Waking up in 10.00 seconds. | |
[2018-02-14 14:29:17,201: DEBUG/Beat] beat: Synchronizing schedule... | |
[2018-02-14 14:29:17,210: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:29:17,218: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->5fa6ebf5-3ea0-4b06-9219-e029f7550c8d | |
[2018-02-14 14:29:17,220: DEBUG/Beat] beat: Waking up in 19.94 seconds. | |
Using /etc/ansible/ansible.cfg as config file | |
127.0.0.1 | SUCCESS => { | |
"changed": false, | |
"elapsed": 0, | |
"path": null, | |
"port": 5432, | |
"search_regex": null, | |
"state": "started" | |
} | |
Using /etc/ansible/ansible.cfg as config file | |
127.0.0.1 | FAILED! => { | |
"changed": false, | |
"elapsed": 300, | |
"msg": "Timeout when waiting for localhost:11211" | |
} | |
Using /etc/ansible/ansible.cfg as config file |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
192.168.122.1 - - [14/Feb/2018:14:35:51 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-" | |
2018/02/14 14:36:08 [error] 25#0: *56 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:08 [warn] 25#0: *56 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:13 [error] 25#0: *56 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:13 [warn] 25#0: *56 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
192.168.122.1 - - [14/Feb/2018:14:36:13 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-" | |
2018/02/14 14:36:28 [error] 25#0: *59 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:28 [warn] 25#0: *59 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:33 [error] 25#0: *59 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:33 [warn] 25#0: *59 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
192.168.122.1 - - [14/Feb/2018:14:36:33 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-" | |
2018/02/14 14:36:56 [error] 25#0: *62 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:36:56 [warn] 25#0: *62 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:37:01 [error] 25#0: *62 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
2018/02/14 14:37:01 [warn] 25#0: *62 upstream server temporarily disabled while reading response header from upstream, client: 192.168.122.1, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "docker.local" | |
192.168.122.1 - - [14/Feb/2018:14:37:01 +0000] "GET /websocket/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0" "-" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using /etc/ansible/ansible.cfg as config file | |
127.0.0.1 | SUCCESS => { | |
"changed": false, | |
"elapsed": 0, | |
"path": null, | |
"port": 5432, | |
"search_regex": null, | |
"state": "started" | |
} | |
Using /etc/ansible/ansible.cfg as config file | |
127.0.0.1 | FAILED! => { | |
"changed": false, | |
"elapsed": 300, | |
"msg": "Timeout when waiting for localhost:11211" | |
} | |
Using /etc/ansible/ansible.cfg as config file | |
127.0.0.1 | FAILED! => { | |
"changed": false, | |
"elapsed": 300, | |
"msg": "Timeout when waiting for localhost:5672" | |
} | |
Using /etc/ansible/ansible.cfg as config file | |
[DEPRECATION WARNING]: The sudo command line option has been deprecated in | |
favor of the "become" command line arguments. This feature will be removed in | |
version 2.6. Deprecation warnings can be disabled by setting | |
deprecation_warnings=False in ansible.cfg. | |
127.0.0.1 | SUCCESS => { | |
"changed": false, | |
"db": "awx" | |
} | |
Operations to perform: | |
Apply all migrations: auth, conf, contenttypes, django_celery_results, main, sessions, sites, social_django, sso, taggit | |
Running migrations: | |
No migrations to apply. | |
Your models have changes that are not yet reflected in a migration, and so won't be applied. | |
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them. | |
Traceback (most recent call last): | |
File "/usr/bin/awx-manage", line 9, in <module> | |
load_entry_point('awx==1.0.3.14', 'console_scripts', 'awx-manage')() | |
File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 109, in manage | |
execute_from_command_line(sys.argv) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line | |
utility.execute() | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute | |
self.fetch_command(subcommand).run_from_argv(self.argv) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv | |
self.execute(*args, **cmd_options) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute | |
output = self.handle(*args, **options) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/shell.py", line 101, in handle | |
exec(sys.stdin.read()) | |
File "<string>", line 1, in <module> | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 170, in create_superuser | |
return self._create_user(username, email, password, **extra_fields) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py", line 153, in _create_user | |
user.save(using=self._db) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 80, in save | |
super(AbstractBaseUser, self).save(*args, **kwargs) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save | |
force_update=force_update, update_fields=update_fields) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base | |
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table | |
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert | |
using=using, raw=raw) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method | |
return getattr(self.get_queryset(), name)(*args, **kwargs) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert | |
return query.get_compiler(using=using).execute_sql(return_id) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1107, in execute_sql | |
cursor.execute(sql, params) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute | |
return self.cursor.execute(sql, params) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__ | |
six.reraise(dj_exc_type, dj_exc_value, traceback) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute | |
return self.cursor.execute(sql, params) | |
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key" | |
DETAIL: Key (username)=(admin) already exists. | |
Instance already registered awx | |
Instance Group already registered tower | |
2018-02-14 14:39:33,079 CRIT Supervisor running as root (no user in config file) | |
Unlinking stale socket /tmp/supervisor.sock | |
2018-02-14 14:39:33,407 INFO RPC interface 'supervisor' initialized | |
2018-02-14 14:39:33,411 CRIT Server 'unix_http_server' running without any HTTP authentication checking | |
2018-02-14 14:39:33,412 INFO supervisord started with pid 164 | |
2018-02-14 14:39:34,414 INFO spawned: 'awx-config-watcher' with pid 167 | |
2018-02-14 14:39:34,415 INFO spawned: 'celery' with pid 168 | |
2018-02-14 14:39:34,416 INFO spawned: 'channels-worker' with pid 169 | |
2018-02-14 14:39:34,432 INFO spawned: 'callback-receiver' with pid 170 | |
2018-02-14 14:39:34,573 INFO exited: awx-config-watcher (exit status 0; not expected) | |
2018-02-14 14:39:35,577 INFO spawned: 'awx-config-watcher' with pid 193 | |
2018-02-14 14:39:35,579 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) | |
2018-02-14 14:39:35,581 INFO success: channels-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) | |
2018-02-14 14:39:35,581 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) | |
2018-02-14 14:39:35,719 INFO exited: awx-config-watcher (exit status 0; not expected) | |
2018-02-14 14:39:37,733 INFO spawned: 'awx-config-watcher' with pid 200 | |
2018-02-14 14:39:37,984 INFO exited: awx-config-watcher (exit status 0; not expected) | |
2018-02-14 14:39:39,306 INFO awx.main.tasks Syncing Schedules | |
2018-02-14 14:39:39,460 - INFO - runworker - Using single-threaded worker. | |
2018-02-14 14:39:39,463 - INFO - runworker - Running worker against channel layer default (asgi_amqp.core.AMQPChannelLayer) | |
2018-02-14 14:39:39,469 - INFO - worker - Listening on channels websocket.connect, websocket.disconnect, websocket.receive | |
2018-02-14 14:39:39,800 DEBUG awx.main.tasks Registering celery routes for celery@awx | |
2018-02-14 14:39:39,808 INFO awx.main.tasks Workers on tower node 'awx' added routes {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} | |
/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is | |
absolutely not recommended! | |
Please specify a different user using the -u option. | |
User information: uid=0 euid=0 gid=0 egid=0 | |
uid=uid, euid=euid, gid=gid, egid=egid, | |
[2018-02-14 14:39:39,832: DEBUG/MainProcess] | Worker: Preparing bootsteps. | |
[2018-02-14 14:39:39,836: DEBUG/MainProcess] | Worker: Building graph... | |
[2018-02-14 14:39:39,838: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer} | |
[2018-02-14 14:39:39,852: DEBUG/MainProcess] | Consumer: Preparing bootsteps. | |
[2018-02-14 14:39:39,853: DEBUG/MainProcess] | Consumer: Building graph... | |
[2018-02-14 14:39:39,866: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Gossip, Tasks, Control, Agent, event loop} | |
2018-02-14 14:39:39,872 WARNING awx.main.tasks Set hostname to celery@awx | |
[2018-02-14 14:39:39,877: DEBUG/MainProcess] | Worker: Starting Hub | |
[2018-02-14 14:39:39,877: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,877: DEBUG/MainProcess] | Worker: Starting Pool | |
[2018-02-14 14:39:39,878: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,878: DEBUG/MainProcess] | Worker: Starting Autoscaler | |
[2018-02-14 14:39:39,878: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,878: DEBUG/MainProcess] | Worker: Starting Beat | |
[2018-02-14 14:39:39,880: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,880: DEBUG/MainProcess] | Worker: Starting Consumer | |
[2018-02-14 14:39:39,881: DEBUG/MainProcess] | Consumer: Starting Connection | |
[2018-02-14 14:39:39,906: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:39,916: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672/awx | |
[2018-02-14 14:39:39,916: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,916: DEBUG/MainProcess] | Consumer: Starting Events | |
[2018-02-14 14:39:39,929: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:39,934: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,936: DEBUG/MainProcess] | Consumer: Starting Heart | |
[2018-02-14 14:39:39,947: DEBUG/MainProcess] using channel_id: 1 | |
[2018-02-14 14:39:39,956: DEBUG/MainProcess] Channel open | |
[2018-02-14 14:39:39,959: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:39,960: DEBUG/MainProcess] | Consumer: Starting Mingle | |
[2018-02-14 14:39:39,960: INFO/MainProcess] mingle: searching for neighbors | |
[2018-02-14 14:39:39,960: DEBUG/MainProcess] using channel_id: 1 | |
[2018-02-14 14:39:39,965: DEBUG/MainProcess] Channel open | |
[2018-02-14 14:39:39,993: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:40,002: DEBUG/MainProcess] using channel_id: 1 | |
[2018-02-14 14:39:40,005: DEBUG/MainProcess] Channel open | |
2018-02-14 14:39:41,010 INFO spawned: 'awx-config-watcher' with pid 206 | |
[2018-02-14 14:39:41,013: INFO/MainProcess] mingle: all alone | |
[2018-02-14 14:39:41,013: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:41,014: DEBUG/MainProcess] | Consumer: Starting Gossip | |
[2018-02-14 14:39:41,014: DEBUG/MainProcess] using channel_id: 2 | |
[2018-02-14 14:39:41,014: DEBUG/MainProcess] Channel open | |
[2018-02-14 14:39:41,022: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:41,022: DEBUG/MainProcess] | Consumer: Starting Tasks | |
[2018-02-14 14:39:41,064: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:41,064: DEBUG/MainProcess] | Consumer: Starting Control | |
[2018-02-14 14:39:41,064: DEBUG/MainProcess] using channel_id: 3 | |
[2018-02-14 14:39:41,069: DEBUG/MainProcess] Channel open | |
[2018-02-14 14:39:41,074: DEBUG/MainProcess] ^-- substep ok | |
[2018-02-14 14:39:41,074: DEBUG/MainProcess] | Consumer: Starting event loop | |
[2018-02-14 14:39:41,074: DEBUG/MainProcess] | Worker: Hub.register Autoscaler... | |
[2018-02-14 14:39:41,075: DEBUG/MainProcess] | Worker: Hub.register Pool... | |
2018-02-14 14:39:41,078 DEBUG awx.main.tasks Configure celeryd queues task on host celery@awx | |
[2018-02-14 14:39:41,130: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:41,138: DEBUG/MainProcess] using channel_id: 1 | |
[2018-02-14 14:39:41,141: DEBUG/MainProcess] Channel open | |
2018-02-14 14:39:41,195 INFO exited: awx-config-watcher (exit status 0; not expected) | |
[2018-02-14 14:39:41,389: INFO/Beat] beat: Starting... | |
2018-02-14 14:39:41,392 INFO gave up: awx-config-watcher entered FATAL state, too many start retries too quickly | |
[2018-02-14 14:39:41,435: DEBUG/Beat] Current schedule: | |
<ScheduleEntry: purge_stdout_files awx.main.tasks.purge_old_stdout_files() <freq: 7.00 days> | |
<ScheduleEntry: tower_scheduler awx.main.tasks.awx_periodic_scheduler() <freq: 30.00 seconds> | |
<ScheduleEntry: authtoken_cleanup awx.main.tasks.cleanup_authtokens() <freq: 30.00 days> | |
<ScheduleEntry: isolated_heartbeat awx.main.tasks.awx_isolated_heartbeat() <freq: 10.00 minutes> | |
<ScheduleEntry: task_manager awx.main.scheduler.tasks.run_task_manager() <freq: 20.00 seconds> | |
<ScheduleEntry: cluster_heartbeat awx.main.tasks.cluster_node_heartbeat() <freq: 1.00 minute> | |
<ScheduleEntry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)> | |
<ScheduleEntry: admin_checks awx.main.tasks.run_administrative_checks() <freq: 30.00 days> | |
[2018-02-14 14:39:41,439: DEBUG/Beat] beat: Ticking with max interval->1.00 minute | |
[2018-02-14 14:39:41,452: DEBUG/Beat] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:41,458: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:39:41,469: DEBUG/Beat] using channel_id: 1 | |
[2018-02-14 14:39:41,472: DEBUG/Beat] Channel open | |
[2018-02-14 14:39:41,476: DEBUG/Beat] beat: Synchronizing schedule... | |
[2018-02-14 14:39:41,480: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->e733dd87-b9a0-4008-adf6-b79941de4089 | |
[2018-02-14 14:39:41,480: DEBUG/Beat] beat: Waking up in 19.95 seconds. | |
2018-02-14 14:39:45,214 INFO awx.main.tasks Workers on tower node 'awx' unsubscribed from queues [] and subscribed to queues [u'tower', 'tower_instance_router', u'awx'] | |
[2018-02-14 14:39:45,219: INFO/MainProcess] celery@awx ready. | |
[2018-02-14 14:39:45,220: DEBUG/MainProcess] basic.qos: prefetch_count->4 | |
[2018-02-14 14:39:45,225: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:0591ceaf-6a23-4930-8284-247c997a1049] | |
[2018-02-14 14:39:45,236: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'tower', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:32db1876-2615-4dd2-82d8-4953ecf14509] | |
[2018-02-14 14:39:45,241: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'tower_instance_router', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:2a823f79-8a31-4bcb-be8f-ecb3fc516f4c] | |
[2018-02-14 14:39:45,243: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'awx', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'f62c88bd-88e9-35bd-ae1e-7d51a63e72cb', u'exchange': u'reply.celery.pidbox'} ticket:e0fc664f-f99e-4f77-8455-7a86e1a8695d] | |
[2018-02-14 14:39:45,352: INFO/MainProcess] Started consuming from tower | |
[2018-02-14 14:39:45,363: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c88ef338-7ca3-40a5-adf6-0916f74add94] expires:[2018-02-14 14:07:36.923698+00:00] | |
[2018-02-14 14:39:45,368: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c88ef338-7ca3-40a5-adf6-0916f74add94] | |
[2018-02-14 14:39:45,413: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[748a8d64-39e1-4957-8b1e-98e793520a21] expires:[2018-02-14 14:07:56.801677+00:00] | |
[2018-02-14 14:39:45,417: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[748a8d64-39e1-4957-8b1e-98e793520a21] | |
[2018-02-14 14:39:45,427: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7dfe6065-c632-464f-83e7-383e87e59ff3] expires:[2018-02-14 14:07:56.925859+00:00] | |
[2018-02-14 14:39:45,427: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7dfe6065-c632-464f-83e7-383e87e59ff3] | |
[2018-02-14 14:39:45,432: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a7529e2e-55aa-49ce-8612-118692fc4ad4] expires:[2018-02-14 14:08:16.923693+00:00] | |
[2018-02-14 14:39:45,433: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a7529e2e-55aa-49ce-8612-118692fc4ad4] | |
[2018-02-14 14:39:45,440: INFO/MainProcess] Started consuming from tower_instance_router | |
[2018-02-14 14:39:45,442: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[043418e1-416b-4526-9f59-7a4569c92af7] expires:[2018-02-14 14:08:26.806176+00:00] | |
[2018-02-14 14:39:45,442: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[043418e1-416b-4526-9f59-7a4569c92af7] | |
[2018-02-14 14:39:45,448: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b6323ddc-4077-4462-b4f9-58871c63384c] expires:[2018-02-14 14:08:36.936883+00:00] | |
[2018-02-14 14:39:45,449: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b6323ddc-4077-4462-b4f9-58871c63384c] | |
[2018-02-14 14:39:45,458: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[0041b78b-462b-4945-b371-d9407467754e] expires:[2018-02-14 14:08:56.815107+00:00] | |
[2018-02-14 14:39:45,458: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[0041b78b-462b-4945-b371-d9407467754e] | |
[2018-02-14 14:39:45,462: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[199917dd-a6e2-4df9-93ae-7653a87aae3e] expires:[2018-02-14 14:08:56.944057+00:00] | |
[2018-02-14 14:39:45,462: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[199917dd-a6e2-4df9-93ae-7653a87aae3e] | |
[2018-02-14 14:39:45,471: INFO/MainProcess] Started consuming from awx | |
[2018-02-14 14:39:45,476: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a9d2355f-5623-42b0-9158-38b0c3c74326] expires:[2018-02-14 14:09:16.945055+00:00] | |
[2018-02-14 14:39:45,478: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a9d2355f-5623-42b0-9158-38b0c3c74326] | |
[2018-02-14 14:39:45,484: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[19a567b2-e253-41c8-8904-7d9c7d9cabd3] expires:[2018-02-14 14:08:56.661030+00:00] | |
[2018-02-14 14:39:45,484: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[19a567b2-e253-41c8-8904-7d9c7d9cabd3] | |
[2018-02-14 14:39:45,493: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ba296241-c082-49f6-985e-a2e9b6d8bf5b] expires:[2018-02-14 14:09:26.814920+00:00] | |
[2018-02-14 14:39:45,496: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ba296241-c082-49f6-985e-a2e9b6d8bf5b] | |
[2018-02-14 14:39:45,503: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0f296241-ece6-48f5-aab6-2cf5142c9aa1] expires:[2018-02-14 14:09:36.941452+00:00] | |
[2018-02-14 14:39:45,504: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0f296241-ece6-48f5-aab6-2cf5142c9aa1] | |
[2018-02-14 14:39:46,019: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7c72448f-e74a-43ac-b019-4ff22779fd54] expires:[2018-02-14 14:09:56.661826+00:00] | |
[2018-02-14 14:39:46,022: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7c72448f-e74a-43ac-b019-4ff22779fd54] | |
[2018-02-14 14:39:46,028: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ed967de9-af53-4a90-b289-706200e2d5e6] expires:[2018-02-14 14:09:56.825003+00:00] | |
[2018-02-14 14:39:46,031: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ed967de9-af53-4a90-b289-706200e2d5e6] | |
[2018-02-14 14:39:46,036: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[6f6737f5-7f8c-400e-aa8c-c2b76f51ddfc] expires:[2018-02-14 14:10:56.665331+00:00] | |
[2018-02-14 14:39:46,039: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[6f6737f5-7f8c-400e-aa8c-c2b76f51ddfc] | |
[2018-02-14 14:39:46,045: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0ec3ec2e-6cda-4d29-a93e-a213522b4947] expires:[2018-02-14 14:09:56.946058+00:00] | |
[2018-02-14 14:39:46,047: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0ec3ec2e-6cda-4d29-a93e-a213522b4947] | |
[2018-02-14 14:39:46,054: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[772a7400-035a-47c2-abc0-8d810e10ae7d] expires:[2018-02-14 14:10:16.956771+00:00] | |
[2018-02-14 14:39:46,057: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[772a7400-035a-47c2-abc0-8d810e10ae7d] | |
[2018-02-14 14:39:46,064: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[b5f632e1-7a45-4fed-b56c-7d36036fa93c] expires:[2018-02-14 14:11:56.662976+00:00] | |
[2018-02-14 14:39:46,064: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[b5f632e1-7a45-4fed-b56c-7d36036fa93c] | |
[2018-02-14 14:39:46,071: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b99d717e-2fde-4d0e-8d97-1050360208f0] expires:[2018-02-14 14:10:26.821710+00:00] | |
[2018-02-14 14:39:46,071: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b99d717e-2fde-4d0e-8d97-1050360208f0] | |
[2018-02-14 14:39:46,079: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[3ce1dcb1-def1-4ba9-af1a-d33e2697de59] expires:[2018-02-14 14:12:56.667788+00:00] | |
[2018-02-14 14:39:46,079: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[3ce1dcb1-def1-4ba9-af1a-d33e2697de59] | |
[2018-02-14 14:39:46,084: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e500e0d2-3722-41bc-807b-d01843a40ba0] expires:[2018-02-14 14:10:36.957260+00:00] | |
[2018-02-14 14:39:46,084: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[e500e0d2-3722-41bc-807b-d01843a40ba0] | |
[2018-02-14 14:39:46,088: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a74b1767-303e-4e69-afec-f4a6853983e7] expires:[2018-02-14 14:10:56.822199+00:00] | |
[2018-02-14 14:39:46,089: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a74b1767-303e-4e69-afec-f4a6853983e7] | |
[2018-02-14 14:39:46,093: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[6c40f218-8c61-4945-ac1d-d9a9aa85bd72] expires:[2018-02-14 14:10:56.955529+00:00] | |
[2018-02-14 14:39:46,093: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[6c40f218-8c61-4945-ac1d-d9a9aa85bd72] | |
[2018-02-14 14:39:46,103: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ce1ed84f-ad34-484c-8dc7-56806bb33a91] expires:[2018-02-14 14:11:16.963834+00:00] | |
[2018-02-14 14:39:46,105: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ce1ed84f-ad34-484c-8dc7-56806bb33a91] | |
[2018-02-14 14:39:46,112: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[80492b24-211e-459f-8656-22c467c6a1b0] expires:[2018-02-14 14:13:56.846595+00:00] | |
[2018-02-14 14:39:46,114: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[80492b24-211e-459f-8656-22c467c6a1b0] | |
[2018-02-14 14:39:46,120: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7310d492-31e6-4b74-a1d8-44fee934be09] expires:[2018-02-14 14:14:56.849140+00:00] | |
[2018-02-14 14:39:46,122: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7310d492-31e6-4b74-a1d8-44fee934be09] | |
[2018-02-14 14:39:46,130: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8174788c-dbb5-43b5-97dc-cba1d28c45d8] expires:[2018-02-14 14:15:56.854597+00:00] | |
[2018-02-14 14:39:46,133: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8174788c-dbb5-43b5-97dc-cba1d28c45d8] | |
[2018-02-14 14:39:46,140: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[51a232c8-ef59-4bf7-a35f-2dffeaf21c26] expires:[2018-02-14 14:16:56.852644+00:00] | |
[2018-02-14 14:39:46,140: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[51a232c8-ef59-4bf7-a35f-2dffeaf21c26] | |
[2018-02-14 14:39:46,149: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[54f44a88-93a6-401b-8a67-1df94bb7bc77] expires:[2018-02-14 14:11:26.826999+00:00] | |
[2018-02-14 14:39:46,149: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[54f44a88-93a6-401b-8a67-1df94bb7bc77] | |
[2018-02-14 14:39:46,155: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[94331b5a-747d-413e-9ff0-64ba8899a643] expires:[2018-02-14 14:17:56.854611+00:00] | |
[2018-02-14 14:39:46,155: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[94331b5a-747d-413e-9ff0-64ba8899a643] | |
[2018-02-14 14:39:46,161: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c88f36d3-f40d-4bf6-8dd5-eb35c44db829] expires:[2018-02-14 14:11:36.978382+00:00] | |
[2018-02-14 14:39:46,162: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c88f36d3-f40d-4bf6-8dd5-eb35c44db829] | |
[2018-02-14 14:39:46,168: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8a87ca2e-7c2e-4bdc-b5d3-4c36141069ff] expires:[2018-02-14 14:18:56.850983+00:00] | |
[2018-02-14 14:39:46,168: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8a87ca2e-7c2e-4bdc-b5d3-4c36141069ff] | |
[2018-02-14 14:39:46,172: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7e3784c4-2caf-4d5f-9bab-6d6ff7ad543b] expires:[2018-02-14 14:19:56.853552+00:00] | |
[2018-02-14 14:39:46,172: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7e3784c4-2caf-4d5f-9bab-6d6ff7ad543b] | |
[2018-02-14 14:39:46,178: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[1787ded5-bd6f-406e-b71a-8a116d69023d] expires:[2018-02-14 14:11:56.834816+00:00] | |
[2018-02-14 14:39:46,180: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[1787ded5-bd6f-406e-b71a-8a116d69023d] | |
[2018-02-14 14:39:46,186: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[05e0edcc-9a78-451f-8e5e-0a36a627d360] expires:[2018-02-14 14:20:56.854025+00:00] | |
[2018-02-14 14:39:46,188: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[05e0edcc-9a78-451f-8e5e-0a36a627d360] | |
[2018-02-14 14:39:46,195: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[8560b004-f821-4138-9519-49241e6ac0d7] expires:[2018-02-14 14:21:56.858278+00:00] | |
[2018-02-14 14:39:46,197: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[8560b004-f821-4138-9519-49241e6ac0d7] | |
[2018-02-14 14:39:46,203: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[c7b0250d-e05f-4a4f-9e4a-c50a95bbad1b] expires:[2018-02-14 14:22:56.856823+00:00] | |
[2018-02-14 14:39:46,203: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[c7b0250d-e05f-4a4f-9e4a-c50a95bbad1b] | |
[2018-02-14 14:39:46,209: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4f5be35b-b8b7-4982-acfe-520ca6609304] expires:[2018-02-14 14:11:56.980504+00:00] | |
[2018-02-14 14:39:46,210: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4f5be35b-b8b7-4982-acfe-520ca6609304] | |
[2018-02-14 14:39:46,217: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4b3bec6f-54b3-4735-be28-70c32d7b9aba] expires:[2018-02-14 14:12:16.975342+00:00] | |
[2018-02-14 14:39:46,217: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4b3bec6f-54b3-4735-be28-70c32d7b9aba] | |
[2018-02-14 14:39:46,225: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[bcc9b244-f2bc-4462-9e0d-12938b49c632] expires:[2018-02-14 14:23:56.963322+00:00] | |
[2018-02-14 14:39:46,225: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[bcc9b244-f2bc-4462-9e0d-12938b49c632] | |
[2018-02-14 14:39:47,449: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[7c5f42b9-0b93-4824-ad04-c27adeed3f6a] expires:[2018-02-14 14:24:56.959598+00:00] | |
[2018-02-14 14:39:47,451: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[7c5f42b9-0b93-4824-ad04-c27adeed3f6a] | |
[2018-02-14 14:39:47,457: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[23168038-1560-416c-be46-c0df71b238d9] expires:[2018-02-14 14:12:26.835610+00:00] | |
[2018-02-14 14:39:47,459: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[23168038-1560-416c-be46-c0df71b238d9] | |
[2018-02-14 14:39:47,466: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[dcb1f885-9a4b-4e14-be00-5a50c60eb4c6] expires:[2018-02-14 14:25:56.964345+00:00] | |
[2018-02-14 14:39:47,468: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[dcb1f885-9a4b-4e14-be00-5a50c60eb4c6] | |
[2018-02-14 14:39:47,475: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[37032744-9082-4fd9-9de7-c9e92ede78c0] expires:[2018-02-14 14:12:36.979865+00:00] | |
[2018-02-14 14:39:47,477: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[37032744-9082-4fd9-9de7-c9e92ede78c0] | |
[2018-02-14 14:39:47,483: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[a7dcf11c-5956-4072-9da7-2f50bb807c61] expires:[2018-02-14 14:26:56.960077+00:00] | |
[2018-02-14 14:39:47,486: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[a7dcf11c-5956-4072-9da7-2f50bb807c61] | |
[2018-02-14 14:39:47,491: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[6837cdc0-0fb0-4e34-9cf6-e786d5ab8cb2] expires:[2018-02-14 14:27:56.966460+00:00] | |
[2018-02-14 14:39:47,492: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[6837cdc0-0fb0-4e34-9cf6-e786d5ab8cb2] | |
[2018-02-14 14:39:47,498: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[f1783d08-e598-4849-bed4-0e56fa4747d2] expires:[2018-02-14 14:12:56.840884+00:00] | |
[2018-02-14 14:39:47,498: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[f1783d08-e598-4849-bed4-0e56fa4747d2] | |
[2018-02-14 14:39:47,504: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[cea98b40-6df1-48a7-8bd3-0c782516d978] expires:[2018-02-14 14:28:56.966765+00:00] | |
[2018-02-14 14:39:47,504: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[cea98b40-6df1-48a7-8bd3-0c782516d978] | |
[2018-02-14 14:39:47,508: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[894fdc2c-584c-44da-b0f7-406e948a57da] expires:[2018-02-14 14:12:56.981899+00:00] | |
[2018-02-14 14:39:47,509: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[894fdc2c-584c-44da-b0f7-406e948a57da] | |
[2018-02-14 14:39:47,515: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[4d6324b3-a5f0-4022-827d-4fb098d535ff] expires:[2018-02-14 14:29:56.964320+00:00] | |
[2018-02-14 14:39:47,516: INFO/MainProcess] Discarding revoked task: awx.main.tasks.cluster_node_heartbeat[4d6324b3-a5f0-4022-827d-4fb098d535ff] | |
[2018-02-14 14:39:47,521: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[bd37a68f-9b2a-4153-ad4d-cbefdc982d5c] expires:[2018-02-14 14:13:16.983728+00:00] | |
[2018-02-14 14:39:47,521: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[bd37a68f-9b2a-4153-ad4d-cbefdc982d5c] | |
[2018-02-14 14:39:47,526: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[3b63c184-b7f5-4c4c-9864-1596961c8a83] expires:[2018-02-14 14:13:26.843203+00:00] | |
[2018-02-14 14:39:47,526: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[3b63c184-b7f5-4c4c-9864-1596961c8a83] | |
[2018-02-14 14:39:47,531: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[d0314510-e072-4e21-a459-78a00c007450] expires:[2018-02-14 14:13:36.982907+00:00] | |
[2018-02-14 14:39:47,531: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[d0314510-e072-4e21-a459-78a00c007450] | |
[2018-02-14 14:39:47,539: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9e09f23d-8c88-4b4b-b0b1-34b1bf6db12c] expires:[2018-02-14 14:13:56.845189+00:00] | |
[2018-02-14 14:39:47,541: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9e09f23d-8c88-4b4b-b0b1-34b1bf6db12c] | |
[2018-02-14 14:39:47,547: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[723e5bce-47f3-4d15-aca1-eb429793cd0d] expires:[2018-02-14 14:13:56.985640+00:00] | |
[2018-02-14 14:39:47,550: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[723e5bce-47f3-4d15-aca1-eb429793cd0d] | |
[2018-02-14 14:39:47,555: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b27fa26e-7638-4468-be8a-2d17625cd8fe] expires:[2018-02-14 14:14:16.992320+00:00] | |
[2018-02-14 14:39:47,558: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b27fa26e-7638-4468-be8a-2d17625cd8fe] | |
[2018-02-14 14:39:47,566: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[e896ddab-d18b-48d6-9095-ebef654b5d25] expires:[2018-02-14 14:14:26.856740+00:00] | |
[2018-02-14 14:39:47,571: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[e896ddab-d18b-48d6-9095-ebef654b5d25] | |
[2018-02-14 14:39:47,582: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8aa0c58a-8130-46a9-bcbf-3822cf5c40be] expires:[2018-02-14 14:14:36.999752+00:00] | |
[2018-02-14 14:39:47,583: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8aa0c58a-8130-46a9-bcbf-3822cf5c40be] | |
[2018-02-14 14:39:47,592: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[131146ff-b714-40ef-9c41-8a40f96d77f6] expires:[2018-02-14 14:14:56.863443+00:00] | |
[2018-02-14 14:39:47,592: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[131146ff-b714-40ef-9c41-8a40f96d77f6] | |
[2018-02-14 14:39:47,599: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[58cb6b31-5686-4d2d-85ff-1ed259cc311d] expires:[2018-02-14 14:14:57.006261+00:00] | |
[2018-02-14 14:39:47,599: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[58cb6b31-5686-4d2d-85ff-1ed259cc311d] | |
[2018-02-14 14:39:47,605: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4f530856-01ac-4b58-a601-b3f68d2bb316] expires:[2018-02-14 14:15:17.002741+00:00] | |
[2018-02-14 14:39:47,605: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4f530856-01ac-4b58-a601-b3f68d2bb316] | |
[2018-02-14 14:39:47,611: INFO/MainProcess] Received task: awx.main.tasks.awx_isolated_heartbeat[98257bcd-61cf-4775-8db7-a83ca2f3152c] expires:[2018-02-14 14:35:06.870074+00:00] | |
[2018-02-14 14:39:47,611: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_isolated_heartbeat[98257bcd-61cf-4775-8db7-a83ca2f3152c] | |
[2018-02-14 14:39:47,617: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a5335e06-943a-417f-8261-e4dda17315e4] expires:[2018-02-14 14:15:26.877808+00:00] | |
[2018-02-14 14:39:47,618: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a5335e06-943a-417f-8261-e4dda17315e4] | |
[2018-02-14 14:39:47,621: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7aa364a9-9b53-4b0a-8ff8-04c0dd77610a] expires:[2018-02-14 14:15:37.004958+00:00] | |
[2018-02-14 14:39:47,622: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7aa364a9-9b53-4b0a-8ff8-04c0dd77610a] | |
[2018-02-14 14:39:47,626: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800] | |
[2018-02-14 14:39:47,626: INFO/MainProcess] Scaling up 1 processes. | |
[2018-02-14 14:39:47,734: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_launch', '8951b093-78dd-4743-bbeb-163851db0800', {'origin': 'gen26@awxweb', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_launch', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '8951b093-78dd-4743-bbeb-163851db0800', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(6,)', 'eta': None, 'parent_id': None, u'reply_to': '75961f62-87ce-37a0-8d47-a1c9e3a1eaf2', 'id': '8951b093-78dd-4743-bbeb-163851db0800', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:47,739: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9e6ae3d8-da80-4466-abd8-f359846d3a18] expires:[2018-02-14 14:15:56.893653+00:00] | |
[2018-02-14 14:39:47,742: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9e6ae3d8-da80-4466-abd8-f359846d3a18] | |
[2018-02-14 14:39:47,750: DEBUG/ForkPoolWorker-2] Closed channel #1 | |
[2018-02-14 14:39:47,754: DEBUG/ForkPoolWorker-2] Closed channel #1 | |
[2018-02-14 14:39:47,834: DEBUG/MainProcess] basic.qos: prefetch_count->8 | |
[2018-02-14 14:39:47,846: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[139091ef-2e34-40a5-b041-cfb8ab0bb637] expires:[2018-02-14 14:15:57.007459+00:00] | |
[2018-02-14 14:39:47,849: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[139091ef-2e34-40a5-b041-cfb8ab0bb637] | |
[2018-02-14 14:39:47,868: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8bb4e51d-8e88-4438-bf75-682ca3505607] expires:[2018-02-14 14:16:17.013451+00:00] | |
[2018-02-14 14:39:47,869: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8bb4e51d-8e88-4438-bf75-682ca3505607] | |
[2018-02-14 14:39:47,879: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[9feca935-218b-4b31-b4fc-34f19fae17d6] expires:[2018-02-14 14:16:26.890092+00:00] | |
[2018-02-14 14:39:47,880: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[9feca935-218b-4b31-b4fc-34f19fae17d6] | |
[2018-02-14 14:39:47,886: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fd9d5959-c3ef-4fc0-b74a-92fd0554d2a1] expires:[2018-02-14 14:16:37.017863+00:00] | |
[2018-02-14 14:39:47,886: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fd9d5959-c3ef-4fc0-b74a-92fd0554d2a1] | |
[2018-02-14 14:39:47,892: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[6385cee9-fe34-4fb0-bb73-8635c211c143] expires:[2018-02-14 14:16:56.903719+00:00] | |
[2018-02-14 14:39:47,892: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[6385cee9-fe34-4fb0-bb73-8635c211c143] | |
[2018-02-14 14:39:47,902: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[acdf26ce-a740-432a-aefb-607708b5c2a3] expires:[2018-02-14 14:16:57.020229+00:00] | |
[2018-02-14 14:39:47,902: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[acdf26ce-a740-432a-aefb-607708b5c2a3] | |
2018-02-14 14:39:47,915 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:39:47,919: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[61e9eb72-863c-4ede-b9cf-d0390f74f015] expires:[2018-02-14 14:17:17.032842+00:00] | |
2018-02-14 14:39:47,922 DEBUG awx.main.scheduler Failing inconsistent running jobs. | |
[2018-02-14 14:39:47,922: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[61e9eb72-863c-4ede-b9cf-d0390f74f015] | |
[2018-02-14 14:39:47,935: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a046f09b-5965-4ccf-97f4-01458f5bf123] expires:[2018-02-14 14:17:26.899325+00:00] | |
[2018-02-14 14:39:47,938: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a046f09b-5965-4ccf-97f4-01458f5bf123] | |
[2018-02-14 14:39:47,944: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8b23cb55-28a8-4bcb-abe0-a3e3957b30c2] expires:[2018-02-14 14:17:37.038289+00:00] | |
[2018-02-14 14:39:47,947: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8b23cb55-28a8-4bcb-abe0-a3e3957b30c2] | |
[2018-02-14 14:39:47,967: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800] pid:207 | |
[2018-02-14 14:39:47,968: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[91866090-16fe-4a61-8541-ca63aa41eedf] expires:[2018-02-14 14:17:56.913881+00:00] | |
[2018-02-14 14:39:47,968: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[91866090-16fe-4a61-8541-ca63aa41eedf] | |
[2018-02-14 14:39:47,982: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[38f9a9fc-562a-4d90-a8e3-6aa6793d42a4] expires:[2018-02-14 14:17:57.042199+00:00] | |
[2018-02-14 14:39:47,985: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[38f9a9fc-562a-4d90-a8e3-6aa6793d42a4] | |
[2018-02-14 14:39:47,989: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:47,992: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2b18e67e-417f-4070-a417-ed6e6aca51be] expires:[2018-02-14 14:18:17.043270+00:00] | |
[2018-02-14 14:39:47,995: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2b18e67e-417f-4070-a417-ed6e6aca51be] | |
[2018-02-14 14:39:47,999: DEBUG/ForkPoolWorker-2] using channel_id: 1 | |
[2018-02-14 14:39:48,008: DEBUG/ForkPoolWorker-2] Channel open | |
[2018-02-14 14:39:48,030: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:48,033: DEBUG/ForkPoolWorker-2] using channel_id: 1 | |
[2018-02-14 14:39:48,037: DEBUG/ForkPoolWorker-2] Channel open | |
[2018-02-14 14:39:48,050: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[5b8d1963-e5e9-4ab4-aae7-6bf590d04654] expires:[2018-02-14 14:18:26.913365+00:00] | |
[2018-02-14 14:39:48,050: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[5b8d1963-e5e9-4ab4-aae7-6bf590d04654] | |
[2018-02-14 14:39:48,059: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[73505add-ba4e-41e1-ae56-f0183a114ea9] expires:[2018-02-14 14:18:37.041372+00:00] | |
[2018-02-14 14:39:48,059: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[73505add-ba4e-41e1-ae56-f0183a114ea9] | |
[2018-02-14 14:39:48,063: DEBUG/MainProcess] pidbox received method active() [reply_to:{u'routing_key': u'a5d25d51-18e0-3571-bb05-aa52c89277ee', u'exchange': u'reply.celery.pidbox'} ticket:6465f4b0-51ec-4c58-b827-a7904a0e3651] | |
[2018-02-14 14:39:48,063: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer') | |
Traceback (most recent call last): | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message | |
self.node.handle_message(body, message) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message | |
return self.dispatch(**body) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch | |
ticket=ticket) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply | |
serializer=self.mailbox.serializer) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply | |
**opts | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish | |
exchange_name, declare, | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish | |
mandatory=mandatory, immediate=immediate, | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 1734, in _basic_publish | |
(0, exchange, routing_key, mandatory, immediate), msg | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method | |
conn.frame_writer(1, self.channel_id, sig, args, content) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame | |
write(view[:offset]) | |
File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write | |
self._write(s) | |
File "/usr/lib64/python2.7/socket.py", line 224, in meth | |
return getattr(self._sock,name)(*args) | |
error: [Errno 104] Connection reset by peer | |
[2018-02-14 14:39:48,067: DEBUG/MainProcess] Closed channel #3 | |
[2018-02-14 14:39:48,067: DEBUG/MainProcess] using channel_id: 3 | |
[2018-02-14 14:39:48,068: DEBUG/MainProcess] Channel open | |
[2018-02-14 14:39:48,072: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[3f6e3d80-0922-4782-b0f6-ac35ae08f2c5] expires:[2018-02-14 14:18:56.924039+00:00] | |
[2018-02-14 14:39:48,073: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[3f6e3d80-0922-4782-b0f6-ac35ae08f2c5] | |
[2018-02-14 14:39:48,083: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2f2e1c91-bb61-4644-a205-7b1614d53e43] expires:[2018-02-14 14:18:57.045288+00:00] | |
[2018-02-14 14:39:48,085: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2f2e1c91-bb61-4644-a205-7b1614d53e43] | |
[2018-02-14 14:39:48,091: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0b835b02-8553-47b9-8c80-d2079a1f666c] expires:[2018-02-14 14:19:17.042000+00:00] | |
[2018-02-14 14:39:48,091: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0b835b02-8553-47b9-8c80-d2079a1f666c] | |
[2018-02-14 14:39:48,098: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b0a5cee7-2f6b-485b-972b-8a44d4c94687] expires:[2018-02-14 14:19:26.921786+00:00] | |
[2018-02-14 14:39:48,099: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b0a5cee7-2f6b-485b-972b-8a44d4c94687] | |
[2018-02-14 14:39:48,105: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[79d2db55-03cd-431f-a2ec-a0769006a02a] expires:[2018-02-14 14:19:37.040522+00:00] | |
[2018-02-14 14:39:48,107: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[79d2db55-03cd-431f-a2ec-a0769006a02a] | |
[2018-02-14 14:39:48,113: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[0642cc88-2d4b-44c1-bbbd-ad3b4a50ab7a] expires:[2018-02-14 14:19:56.926535+00:00] | |
[2018-02-14 14:39:48,114: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[0642cc88-2d4b-44c1-bbbd-ad3b4a50ab7a] | |
[2018-02-14 14:39:48,121: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fba37cec-3389-4253-b979-26327ff28d40] expires:[2018-02-14 14:19:57.043013+00:00] | |
[2018-02-14 14:39:48,121: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fba37cec-3389-4253-b979-26327ff28d40] | |
[2018-02-14 14:39:48,126: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[aa5b8b07-7754-4021-9ecf-1ac9230e3328] expires:[2018-02-14 14:20:17.054016+00:00] | |
[2018-02-14 14:39:48,127: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[aa5b8b07-7754-4021-9ecf-1ac9230e3328] | |
[2018-02-14 14:39:48,134: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[2f5ced45-c722-410e-84ef-f71efa60fce8] expires:[2018-02-14 14:20:26.923924+00:00] | |
[2018-02-14 14:39:48,135: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[2f5ced45-c722-410e-84ef-f71efa60fce8] | |
[2018-02-14 14:39:48,139: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[48621839-d6f0-483d-b4a4-df5c320a5bf5] expires:[2018-02-14 14:20:37.058344+00:00] | |
[2018-02-14 14:39:48,139: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[48621839-d6f0-483d-b4a4-df5c320a5bf5] | |
[2018-02-14 14:39:48,144: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[2796819f-e082-4e02-80b8-1dcd57048ea7] expires:[2018-02-14 14:20:56.937793+00:00] | |
[2018-02-14 14:39:48,144: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[2796819f-e082-4e02-80b8-1dcd57048ea7] | |
[2018-02-14 14:39:48,149: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4a00ef89-15e3-4d63-b17c-00e4fc903c46] expires:[2018-02-14 14:20:57.064591+00:00] | |
[2018-02-14 14:39:48,149: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[4a00ef89-15e3-4d63-b17c-00e4fc903c46] | |
[2018-02-14 14:39:48,153: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[b0ff083a-3867-443e-8b4d-d59c0a312f54] expires:[2018-02-14 14:21:17.065031+00:00] | |
[2018-02-14 14:39:48,154: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[b0ff083a-3867-443e-8b4d-d59c0a312f54] | |
[2018-02-14 14:39:48,157: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[a77a9add-d0b3-41e4-bae6-c2cb07a64e75] expires:[2018-02-14 14:21:26.934691+00:00] | |
[2018-02-14 14:39:48,158: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[a77a9add-d0b3-41e4-bae6-c2cb07a64e75] | |
[2018-02-14 14:39:48,164: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c84b0b87-ff9b-4534-a33c-4a830a5af044] expires:[2018-02-14 14:21:37.060661+00:00] | |
[2018-02-14 14:39:48,167: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c84b0b87-ff9b-4534-a33c-4a830a5af044] | |
[2018-02-14 14:39:48,175: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[6a0dbc1e-1b72-4fc1-a342-1dad67131d5c] expires:[2018-02-14 14:21:56.945838+00:00] | |
[2018-02-14 14:39:48,180: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[6a0dbc1e-1b72-4fc1-a342-1dad67131d5c] | |
[2018-02-14 14:39:48,190: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[2f8d0048-7fd0-4a34-bfa5-410aa8ed16f2] expires:[2018-02-14 14:21:57.067142+00:00] | |
[2018-02-14 14:39:48,190: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[2f8d0048-7fd0-4a34-bfa5-410aa8ed16f2] | |
[2018-02-14 14:39:48,199: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8a683299-debd-4fee-96fc-22bdbcd15d9f] expires:[2018-02-14 14:22:17.072385+00:00] | |
[2018-02-14 14:39:48,199: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8a683299-debd-4fee-96fc-22bdbcd15d9f] | |
[2018-02-14 14:39:48,207: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[731aa059-b04a-429a-837e-99a42650e474] expires:[2018-02-14 14:22:26.942150+00:00] | |
[2018-02-14 14:39:48,207: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[731aa059-b04a-429a-837e-99a42650e474] | |
[2018-02-14 14:39:48,213: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[9ad2e84f-543c-4476-baae-4e1f36546a3b] expires:[2018-02-14 14:22:37.079048+00:00] | |
[2018-02-14 14:39:48,214: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[9ad2e84f-543c-4476-baae-4e1f36546a3b] | |
[2018-02-14 14:39:48,218: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[e11c265c-d145-492e-97a4-c71b4617a340] expires:[2018-02-14 14:22:56.945959+00:00] | |
[2018-02-14 14:39:48,218: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[e11c265c-d145-492e-97a4-c71b4617a340] | |
[2018-02-14 14:39:48,222: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[fedc34b3-9d6f-4688-93b8-c0ae3b47d7e4] expires:[2018-02-14 14:22:57.076440+00:00] | |
[2018-02-14 14:39:48,223: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[fedc34b3-9d6f-4688-93b8-c0ae3b47d7e4] | |
[2018-02-14 14:39:48,230: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f61599a0-ff55-4c57-920f-e3aecb0a48e3] expires:[2018-02-14 14:23:17.090884+00:00] | |
[2018-02-14 14:39:48,231: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f61599a0-ff55-4c57-920f-e3aecb0a48e3] | |
[2018-02-14 14:39:48,239: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[1049bd07-4015-41c7-8130-1412d7a2783d] expires:[2018-02-14 14:23:26.947478+00:00] | |
[2018-02-14 14:39:48,239: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[1049bd07-4015-41c7-8130-1412d7a2783d] | |
[2018-02-14 14:39:48,245: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a69ab1af-1fd3-4b51-9fe8-b1f67fc9f598] expires:[2018-02-14 14:23:37.104587+00:00] | |
[2018-02-14 14:39:48,248: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a69ab1af-1fd3-4b51-9fe8-b1f67fc9f598] | |
[2018-02-14 14:39:48,254: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[cc3d218d-ddd8-42ad-bfb6-50ffd675637b] expires:[2018-02-14 14:23:56.945156+00:00] | |
[2018-02-14 14:39:48,254: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[cc3d218d-ddd8-42ad-bfb6-50ffd675637b] | |
[2018-02-14 14:39:48,260: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[40442b38-c59b-44d3-99bb-64e0abd72f07] expires:[2018-02-14 14:23:57.106369+00:00] | |
[2018-02-14 14:39:48,263: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[40442b38-c59b-44d3-99bb-64e0abd72f07] | |
[2018-02-14 14:39:48,268: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f7d29095-7dfc-4a87-9eb7-5b127498bf6b] expires:[2018-02-14 14:24:17.116393+00:00] | |
[2018-02-14 14:39:48,271: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f7d29095-7dfc-4a87-9eb7-5b127498bf6b] | |
[2018-02-14 14:39:48,284: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[90e29f26-9d4b-4170-9f66-828e0ca4294a] expires:[2018-02-14 14:24:26.946368+00:00] | |
[2018-02-14 14:39:48,284: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[90e29f26-9d4b-4170-9f66-828e0ca4294a] | |
[2018-02-14 14:39:48,290: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[f93a2799-a1ce-4852-84b6-908a1084b767] expires:[2018-02-14 14:24:37.119256+00:00] | |
[2018-02-14 14:39:48,290: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[f93a2799-a1ce-4852-84b6-908a1084b767] | |
[2018-02-14 14:39:48,295: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[970a32e8-d01c-4c1e-a241-42645a6ae86e] expires:[2018-02-14 14:24:56.947182+00:00] | |
[2018-02-14 14:39:48,295: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[970a32e8-d01c-4c1e-a241-42645a6ae86e] | |
[2018-02-14 14:39:48,299: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[0b2a7d19-37dc-4882-b7a5-c74723261cc8] expires:[2018-02-14 14:24:57.119025+00:00] | |
[2018-02-14 14:39:48,300: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[0b2a7d19-37dc-4882-b7a5-c74723261cc8] | |
[2018-02-14 14:39:48,303: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[af5630bc-039f-40df-9653-de0a4c4ded13] expires:[2018-02-14 14:25:17.123895+00:00] | |
[2018-02-14 14:39:48,304: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[af5630bc-039f-40df-9653-de0a4c4ded13] | |
[2018-02-14 14:39:48,310: INFO/MainProcess] Received task: awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] expires:[2018-02-14 14:45:06.870985+00:00] | |
[2018-02-14 14:39:48,312: INFO/MainProcess] Scaling up 1 processes. | |
[2018-02-14 14:39:48,414: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_isolated_heartbeat', 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_isolated_heartbeat', 'group': None, 'root_id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:45:06.870985+00:00', u'correlation_id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': 'd4c178d7-71ce-36c2-a67c-abc95292f38d', 'id': 'a7047bd4-dfdf-4185-8962-7024a09f9cd7', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:48,419: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[4e56d0ca-6990-44bc-8592-6c73d8440a58] expires:[2018-02-14 14:25:26.980125+00:00] | |
[2018-02-14 14:39:48,421: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[4e56d0ca-6990-44bc-8592-6c73d8440a58] | |
[2018-02-14 14:39:48,521: DEBUG/MainProcess] basic.qos: prefetch_count->12 | |
[2018-02-14 14:39:48,529: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[7faf048b-ecce-4e78-bba1-719ebca166df] expires:[2018-02-14 14:25:37.127133+00:00] | |
[2018-02-14 14:39:48,535: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[7faf048b-ecce-4e78-bba1-719ebca166df] | |
[2018-02-14 14:39:48,556: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[74bcb58b-9d46-4633-867e-3894b3ca8ae6] expires:[2018-02-14 14:25:56.991142+00:00] | |
[2018-02-14 14:39:48,556: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[74bcb58b-9d46-4633-867e-3894b3ca8ae6] | |
[2018-02-14 14:39:48,562: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a67ac9e1-a9f0-4aab-af49-096fee1ab3d7] expires:[2018-02-14 14:25:57.128793+00:00] | |
[2018-02-14 14:39:48,563: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a67ac9e1-a9f0-4aab-af49-096fee1ab3d7] | |
[2018-02-14 14:39:48,568: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ab1cf795-3def-4b60-a5de-09ecefd1e2ea] expires:[2018-02-14 14:26:17.137292+00:00] | |
[2018-02-14 14:39:48,568: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ab1cf795-3def-4b60-a5de-09ecefd1e2ea] | |
[2018-02-14 14:39:48,577: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[d7b9d8ce-a39c-4809-a041-c4f58b1efb50] expires:[2018-02-14 14:26:26.990658+00:00] | |
[2018-02-14 14:39:48,578: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[d7b9d8ce-a39c-4809-a041-c4f58b1efb50] | |
[2018-02-14 14:39:48,584: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[c693eedc-0daa-4da6-9575-ec7de6a1dba4] expires:[2018-02-14 14:26:37.136958+00:00] | |
[2018-02-14 14:39:48,584: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[c693eedc-0daa-4da6-9575-ec7de6a1dba4] | |
[2018-02-14 14:39:48,591: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[69128d40-77e3-480c-a4ce-f4c40a98e699] expires:[2018-02-14 14:26:57.001989+00:00] | |
[2018-02-14 14:39:48,591: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[69128d40-77e3-480c-a4ce-f4c40a98e699] | |
[2018-02-14 14:39:48,598: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[806b8cce-d9d8-486d-8bc2-ff48a512f092] expires:[2018-02-14 14:26:57.142839+00:00] | |
[2018-02-14 14:39:48,598: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[806b8cce-d9d8-486d-8bc2-ff48a512f092] | |
[2018-02-14 14:39:48,603: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e453574b-ecf2-46f8-b631-0feacd52628c] expires:[2018-02-14 14:27:17.153026+00:00] | |
[2018-02-14 14:39:48,603: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[e453574b-ecf2-46f8-b631-0feacd52628c] | |
[2018-02-14 14:39:48,609: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ec75781c-cb03-4d28-9770-37d431102560] expires:[2018-02-14 14:27:26.997756+00:00] | |
[2018-02-14 14:39:48,609: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[ec75781c-cb03-4d28-9770-37d431102560] | |
[2018-02-14 14:39:48,618: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125] | |
[2018-02-14 14:39:48,619: INFO/MainProcess] Scaling up 1 processes. | |
2018-02-14 14:39:48,619 DEBUG awx.main.tasks Controlling node checking for any isolated management tasks. | |
[2018-02-14 14:39:48,645: INFO/ForkPoolWorker-3] Task awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] succeeded in 0.0323591410006s: None | |
[2018-02-14 14:39:48,763: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_launch', '323f9e43-4f90-4efe-bec0-a46c60326125', {'origin': 'gen26@awxweb', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_launch', 'group': None, 'root_id': '323f9e43-4f90-4efe-bec0-a46c60326125', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '323f9e43-4f90-4efe-bec0-a46c60326125', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(7,)', 'eta': None, 'parent_id': None, u'reply_to': '75961f62-87ce-37a0-8d47-a1c9e3a1eaf2', 'id': '323f9e43-4f90-4efe-bec0-a46c60326125', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:48,771: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[ee1642c3-c826-47ee-a1bd-7ac7558919cd] expires:[2018-02-14 14:27:37.162120+00:00] | |
[2018-02-14 14:39:48,771: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[ee1642c3-c826-47ee-a1bd-7ac7558919cd] | |
[2018-02-14 14:39:48,943: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[a3d03326-ca82-41d4-ab26-d9cff55bab44] expires:[2018-02-14 14:27:57.151092+00:00] | |
[2018-02-14 14:39:48,950: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[a3d03326-ca82-41d4-ab26-d9cff55bab44] | |
[2018-02-14 14:39:48,977: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[45b096be-28ff-4395-8c40-027b55d4a0d3] expires:[2018-02-14 14:27:57.159716+00:00] | |
[2018-02-14 14:39:48,977: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[45b096be-28ff-4395-8c40-027b55d4a0d3] | |
[2018-02-14 14:39:48,985: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[6a97de86-4e3b-4f7d-aa97-c3eda0e423d7] expires:[2018-02-14 14:28:17.160192+00:00] | |
[2018-02-14 14:39:48,985: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[6a97de86-4e3b-4f7d-aa97-c3eda0e423d7] | |
[2018-02-14 14:39:48,993: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[4d2069fc-f816-4088-af28-a37bf2dbac00] expires:[2018-02-14 14:28:27.159909+00:00] | |
[2018-02-14 14:39:48,995: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[4d2069fc-f816-4088-af28-a37bf2dbac00] | |
[2018-02-14 14:39:49,000: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_isolated_heartbeat[a7047bd4-dfdf-4185-8962-7024a09f9cd7] pid:208 | |
[2018-02-14 14:39:49,001: DEBUG/MainProcess] basic.qos: prefetch_count->16 | |
[2018-02-14 14:39:49,004: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[d4efc552-6f69-4123-8ffe-97c7fe098370] expires:[2018-02-14 14:28:37.169903+00:00] | |
[2018-02-14 14:39:49,005: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[d4efc552-6f69-4123-8ffe-97c7fe098370] | |
[2018-02-14 14:39:49,012: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[b83ed07d-5c7f-4f56-9269-753bb886b6a6] expires:[2018-02-14 14:28:57.176527+00:00] | |
[2018-02-14 14:39:49,012: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[b83ed07d-5c7f-4f56-9269-753bb886b6a6] | |
[2018-02-14 14:39:49,017: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[8bf5744c-d993-4ce6-94cc-d8675996c20b] expires:[2018-02-14 14:28:57.194499+00:00] | |
[2018-02-14 14:39:49,017: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[8bf5744c-d993-4ce6-94cc-d8675996c20b] | |
[2018-02-14 14:39:49,023: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[cd9c89c9-7d8e-4ea2-ab24-78e8180ab63c] expires:[2018-02-14 14:29:17.205705+00:00] | |
[2018-02-14 14:39:49,023: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[cd9c89c9-7d8e-4ea2-ab24-78e8180ab63c] | |
[2018-02-14 14:39:49,028: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[eaae06c3-5ec9-479e-9640-8e2f26682e69] expires:[2018-02-14 14:29:27.178238+00:00] | |
[2018-02-14 14:39:49,028: INFO/MainProcess] Discarding revoked task: awx.main.tasks.awx_periodic_scheduler[eaae06c3-5ec9-479e-9640-8e2f26682e69] | |
[2018-02-14 14:39:49,035: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[5fa6ebf5-3ea0-4b06-9219-e029f7550c8d] expires:[2018-02-14 14:29:37.212391+00:00] | |
[2018-02-14 14:39:49,035: INFO/MainProcess] Discarding revoked task: awx.main.scheduler.tasks.run_task_manager[5fa6ebf5-3ea0-4b06-9219-e029f7550c8d] | |
[2018-02-14 14:39:49,043: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] expires:[2018-02-14 14:40:01.459842+00:00] | |
2018-02-14 14:39:49,046 ERROR awx.main.scheduler Failed to retrieve active tasks from celery | |
[2018-02-14 14:39:49,052: INFO/MainProcess] Scaling up 1 processes. | |
2018-02-14 14:39:49,166 DEBUG awx.main.scheduler Starting project_update 6 (pending) in group tower (remaining_capacity=4) | |
2018-02-14 14:39:49,197 INFO awx.main.scheduler Submitting project_update 6 (waiting) to instance group 1. | |
2018-02-14 14:39:49,245 DEBUG awx.main.scheduler project_update 6 (waiting) consumed 1 capacity units from tower with prior total of 0 | |
2018-02-14 14:39:49,253 DEBUG awx.main.scheduler project_update 7 (pending) is blocked from running | |
[2018-02-14 14:39:49,262: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'e733dd87-b9a0-4008-adf6-b79941de4089', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'e733dd87-b9a0-4008-adf6-b79941de4089', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:01.459842+00:00', u'correlation_id': 'e733dd87-b9a0-4008-adf6-b79941de4089', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'e733dd87-b9a0-4008-adf6-b79941de4089', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:49,290: DEBUG/MainProcess] basic.qos: prefetch_count->20 | |
[2018-02-14 14:39:49,301: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] pid:208 | |
[2018-02-14 14:39:49,302: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125] pid:209 | |
2018-02-14 14:39:49,359 DEBUG awx.main.scheduler Running Tower task manager. | |
2018-02-14 14:39:49,398 DEBUG awx.main.scheduler Starting Scheduler | |
2018-02-14 14:39:49,403 DEBUG awx.main.scheduler Not running scheduler, another task holds lock | |
[2018-02-14 14:39:49,414: INFO/ForkPoolWorker-4] Task awx.main.scheduler.tasks.run_job_launch[323f9e43-4f90-4efe-bec0-a46c60326125] succeeded in 0.14597921s: None | |
[2018-02-14 14:39:49,430: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:49,434: DEBUG/ForkPoolWorker-2] using channel_id: 1 | |
[2018-02-14 14:39:49,438: DEBUG/ForkPoolWorker-2] Channel open | |
[2018-02-14 14:39:49,455: INFO/MainProcess] Received task: awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31] | |
[2018-02-14 14:39:49,455: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.run_project_update', '415acd10-4f91-43f0-8ef1-c9686b2fbc31', {'origin': 'gen207@awx', 'lang': 'py', 'task': 'awx.main.tasks.run_project_update', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '[6]', 'eta': None, 'parent_id': '8951b093-78dd-4743-bbeb-163851db0800', u'reply_to': '76df055f-4dd9-3f0d-9c89-9acd797180be', 'id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_success", "subtask_type": null, "kwargs": {"task_actual": {"type": "project_update", "id": 6}}, "args": [], "options": {}, "immutable": false}], "errbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_error", "subtask_type": null, "kwargs": {"subtasks":... kwargs:{}) | |
[2018-02-14 14:39:49,463: DEBUG/MainProcess] Task accepted: awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31] pid:210 | |
[2018-02-14 14:39:49,487: INFO/ForkPoolWorker-2] Task awx.main.scheduler.tasks.run_job_launch[8951b093-78dd-4743-bbeb-163851db0800] succeeded in 1.590700005s: None | |
2018-02-14 14:39:49,529 DEBUG awx.main.scheduler project_update 7 (pending) is blocked from running | |
[2018-02-14 14:39:49,558: INFO/ForkPoolWorker-3] Task awx.main.scheduler.tasks.run_task_manager[e733dd87-b9a0-4008-adf6-b79941de4089] succeeded in 0.287292411001s: None | |
[2018-02-14 14:39:49,581: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:49,584: DEBUG/ForkPoolWorker-5] using channel_id: 1 | |
[2018-02-14 14:39:49,588: DEBUG/ForkPoolWorker-5] Channel open | |
[2018-02-14 14:39:49,596: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:49,599: DEBUG/ForkPoolWorker-5] using channel_id: 1 | |
[2018-02-14 14:39:49,603: DEBUG/ForkPoolWorker-5] Channel open | |
[2018-02-14 14:39:51,142: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:51,148: DEBUG/ForkPoolWorker-5] using channel_id: 1 | |
[2018-02-14 14:39:51,157: DEBUG/ForkPoolWorker-5] Channel open | |
2018-02-14 14:39:55,915 INFO awx.main.tasks project_update 6 (running) finished running, producing 40 events. | |
[2018-02-14 14:39:56,095: INFO/MainProcess] Received task: awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888] | |
[2018-02-14 14:39:56,098: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_work_success', '8072962d-7fbe-45b4-beca-e9664a46b888', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_work_success', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '8072962d-7fbe-45b4-beca-e9664a46b888', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(None,)', 'eta': None, 'parent_id': '415acd10-4f91-43f0-8ef1-c9686b2fbc31', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': '8072962d-7fbe-45b4-beca-e9664a46b888', 'kwargsrepr': "{'task_actual': {'type': 'project_update', 'id': 6}}"}, u'[[null], {"task_actual": {"type": "project_update", "id": 6}}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:56,114: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888] pid:208 | |
[2018-02-14 14:39:56,147: INFO/ForkPoolWorker-5] Task awx.main.tasks.run_project_update[415acd10-4f91-43f0-8ef1-c9686b2fbc31] succeeded in 6.686883101s: None | |
[2018-02-14 14:39:56,219: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:56,221: DEBUG/ForkPoolWorker-3] using channel_id: 1 | |
[2018-02-14 14:39:56,226: DEBUG/ForkPoolWorker-3] Channel open | |
[2018-02-14 14:39:56,231: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e] | |
[2018-02-14 14:39:56,231: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_complete', 'da44f23a-f322-4f21-860e-a2ec0f8c843e', {'origin': 'gen208@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_complete', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(6,)', 'eta': None, 'parent_id': '8072962d-7fbe-45b4-beca-e9664a46b888', u'reply_to': 'ce922c0b-ed65-3190-8808-a32fdcd4136b', 'id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', 'kwargsrepr': '{}'}, u'[[6], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:39:56,234: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e] pid:210 | |
[2018-02-14 14:39:56,249: INFO/ForkPoolWorker-3] Task awx.main.tasks.handle_work_success[8072962d-7fbe-45b4-beca-e9664a46b888] succeeded in 0.136103733s: None | |
2018-02-14 14:39:56,273 DEBUG awx.main.scheduler Starting Scheduler | |
2018-02-14 14:39:56,352 DEBUG awx.main.scheduler Starting project_update 7 (pending) in group tower (remaining_capacity=4) | |
2018-02-14 14:39:56,361 INFO awx.main.scheduler Submitting project_update 7 (waiting) to instance group 1. | |
2018-02-14 14:39:56,425 DEBUG awx.main.scheduler project_update 7 (waiting) consumed 1 capacity units from tower with prior total of 0 | |
[2018-02-14 14:39:56,461: INFO/MainProcess] Received task: awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4] | |
[2018-02-14 14:39:56,461: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.run_project_update', '18324028-a169-4f04-a943-c5b331544df4', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.run_project_update', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '18324028-a169-4f04-a943-c5b331544df4', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '[7]', 'eta': None, 'parent_id': 'da44f23a-f322-4f21-860e-a2ec0f8c843e', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': '18324028-a169-4f04-a943-c5b331544df4', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_success", "subtask_type": null, "kwargs": {"task_actual": {"type": "project_update", "id": 7}}, "args": [], "options": {}, "immutable": false}], "errbacks": [{"chord_size": null, "task": "awx.main.tasks.handle_work_error", "subtask_type": null, "kwargs": {"subtasks":... kwargs:{}) | |
[2018-02-14 14:39:56,473: DEBUG/MainProcess] Task accepted: awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4] pid:208 | |
[2018-02-14 14:39:56,501: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_job_complete[da44f23a-f322-4f21-860e-a2ec0f8c843e] succeeded in 0.269563346s: None | |
[2018-02-14 14:39:56,642: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:56,647: DEBUG/ForkPoolWorker-3] using channel_id: 1 | |
[2018-02-14 14:39:56,652: DEBUG/ForkPoolWorker-3] Channel open | |
[2018-02-14 14:39:58,499: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:39:58,506: DEBUG/ForkPoolWorker-3] using channel_id: 1 | |
[2018-02-14 14:39:58,511: DEBUG/ForkPoolWorker-3] Channel open | |
[2018-02-14 14:40:01,075: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:01,075: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 192/515, monotonic - 5233.07401304, last_heartbeat_sent - 5233.07400563, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:01,485: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:40:01,547: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] expires:[2018-02-14 14:40:21.529910+00:00] | |
[2018-02-14 14:40:01,551: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '540dd27b-eb95-442a-9a2b-b8edaa23ddac', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:21.529910+00:00', u'correlation_id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '540dd27b-eb95-442a-9a2b-b8edaa23ddac', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:01,551: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->540dd27b-eb95-442a-9a2b-b8edaa23ddac | |
[2018-02-14 14:40:01,560: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler) | |
[2018-02-14 14:40:01,564: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] pid:210 | |
2018-02-14 14:40:01,572 DEBUG awx.main.scheduler Running Tower task manager. | |
[2018-02-14 14:40:01,571: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->ef10b988-3826-448a-a2d1-82d787821018 | |
[2018-02-14 14:40:01,578: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] expires:[2018-02-14 14:40:21.568358+00:00] | |
[2018-02-14 14:40:01,583: DEBUG/Beat] beat: Waking up in 19.88 seconds. | |
[2018-02-14 14:40:01,586: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', 'ef10b988-3826-448a-a2d1-82d787821018', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': 'ef10b988-3826-448a-a2d1-82d787821018', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:21.568358+00:00', u'correlation_id': 'ef10b988-3826-448a-a2d1-82d787821018', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'ef10b988-3826-448a-a2d1-82d787821018', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
2018-02-14 14:40:01,599 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:40:01,626: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] pid:207 | |
[2018-02-14 14:40:01,666: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[540dd27b-eb95-442a-9a2b-b8edaa23ddac] succeeded in 0.111874957s: None | |
2018-02-14 14:40:01,726 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:07:06.819513+00:00 | |
2018-02-14 14:40:01,755 INFO awx.main.tasks project_update 7 (running) finished running, producing 40 events. | |
[2018-02-14 14:40:01,799: INFO/ForkPoolWorker-2] Task awx.main.tasks.awx_periodic_scheduler[ef10b988-3826-448a-a2d1-82d787821018] succeeded in 0.160589495s: None | |
[2018-02-14 14:40:01,901: INFO/MainProcess] Received task: awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7] | |
[2018-02-14 14:40:01,902: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_work_success', '09395a96-635d-4f4f-aa1f-1e8209828be7', {'origin': 'gen208@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_work_success', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '09395a96-635d-4f4f-aa1f-1e8209828be7', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(None,)', 'eta': None, 'parent_id': '18324028-a169-4f04-a943-c5b331544df4', u'reply_to': 'ce922c0b-ed65-3190-8808-a32fdcd4136b', 'id': '09395a96-635d-4f4f-aa1f-1e8209828be7', 'kwargsrepr': "{'task_actual': {'type': 'project_update', 'id': 7}}"}, u'[[null], {"task_actual": {"type": "project_update", "id": 7}}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:01,907: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7] pid:207 | |
[2018-02-14 14:40:01,936: INFO/ForkPoolWorker-3] Task awx.main.tasks.run_project_update[18324028-a169-4f04-a943-c5b331544df4] succeeded in 5.46970923s: None | |
[2018-02-14 14:40:02,063: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977] | |
[2018-02-14 14:40:02,064: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_job_complete', '6ad5014e-869d-400a-9e38-6de724254977', {'origin': 'gen207@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_job_complete', 'group': None, 'root_id': '8951b093-78dd-4743-bbeb-163851db0800', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': None, u'correlation_id': '6ad5014e-869d-400a-9e38-6de724254977', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(7,)', 'eta': None, 'parent_id': '09395a96-635d-4f4f-aa1f-1e8209828be7', u'reply_to': '3cdbea91-bc1e-3caf-b23a-f02313f7d41c', 'id': '6ad5014e-869d-400a-9e38-6de724254977', 'kwargsrepr': '{}'}, u'[[7], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:02,071: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977] pid:209 | |
[2018-02-14 14:40:02,093: INFO/ForkPoolWorker-2] Task awx.main.tasks.handle_work_success[09395a96-635d-4f4f-aa1f-1e8209828be7] succeeded in 0.181390193s: None | |
2018-02-14 14:40:02,125 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:40:02,175: INFO/ForkPoolWorker-4] Task awx.main.scheduler.tasks.run_job_complete[6ad5014e-869d-400a-9e38-6de724254977] succeeded in 0.0991270140003s: None | |
[2018-02-14 14:40:21,075: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:21,080: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 192/515, now - 196/557, monotonic - 5253.07912095, last_heartbeat_sent - 5253.07909177, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:21,490: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:40:21,499: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->bf9f9083-6225-4ef1-971c-bf89289870f6 | |
[2018-02-14 14:40:21,502: DEBUG/Beat] beat: Waking up in 10.05 seconds. | |
[2018-02-14 14:40:21,502: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] expires:[2018-02-14 14:40:41.492106+00:00] | |
[2018-02-14 14:40:21,503: INFO/MainProcess] Scaling down 3 processes. | |
[2018-02-14 14:40:21,506: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'bf9f9083-6225-4ef1-971c-bf89289870f6', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:41.492106+00:00', u'correlation_id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'bf9f9083-6225-4ef1-971c-bf89289870f6', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:21,506: DEBUG/MainProcess] basic.qos: prefetch_count->8 | |
[2018-02-14 14:40:21,597: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] pid:210 | |
2018-02-14 14:40:21,620 DEBUG awx.main.scheduler Running Tower task manager. | |
2018-02-14 14:40:21,631 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:40:21,653: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[bf9f9083-6225-4ef1-971c-bf89289870f6] succeeded in 0.0604734569997s: None | |
[2018-02-14 14:40:31,560: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler) | |
[2018-02-14 14:40:31,571: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] expires:[2018-02-14 14:40:51.566258+00:00] | |
[2018-02-14 14:40:31,576: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', '8ec215d7-a4e7-4938-8968-52a1fcc492f2', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:40:51.566258+00:00', u'correlation_id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '8ec215d7-a4e7-4938-8968-52a1fcc492f2', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:31,589: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] pid:210 | |
[2018-02-14 14:40:31,593: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->8ec215d7-a4e7-4938-8968-52a1fcc492f2 | |
[2018-02-14 14:40:31,599: INFO/Beat] Scheduler: Sending due task cluster_heartbeat (awx.main.tasks.cluster_node_heartbeat) | |
2018-02-14 14:40:31,606 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:40:01.716576+00:00 | |
[2018-02-14 14:40:31,617: INFO/MainProcess] Received task: awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] expires:[2018-02-14 14:41:21.605482+00:00] | |
[2018-02-14 14:40:31,618: INFO/MainProcess] Scaling up 1 processes. | |
[2018-02-14 14:40:31,616: DEBUG/Beat] awx.main.tasks.cluster_node_heartbeat sent. id->13957fad-e323-47fa-993a-2313a6463bd0 | |
[2018-02-14 14:40:31,623: DEBUG/Beat] beat: Waking up in 9.86 seconds. | |
[2018-02-14 14:40:31,641: INFO/ForkPoolWorker-5] Task awx.main.tasks.awx_periodic_scheduler[8ec215d7-a4e7-4938-8968-52a1fcc492f2] succeeded in 0.0645465070002s: None | |
[2018-02-14 14:40:31,788: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.cluster_node_heartbeat', '13957fad-e323-47fa-993a-2313a6463bd0', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.cluster_node_heartbeat', 'group': None, 'root_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'awx', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.605482+00:00', u'correlation_id': '13957fad-e323-47fa-993a-2313a6463bd0', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '13957fad-e323-47fa-993a-2313a6463bd0', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:31,792: DEBUG/MainProcess] basic.qos: prefetch_count->12 | |
[2018-02-14 14:40:31,803: DEBUG/MainProcess] Task accepted: awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] pid:210 | |
2018-02-14 14:40:31,806 DEBUG awx.main.tasks Cluster node heartbeat task. | |
[2018-02-14 14:40:31,820: INFO/MainProcess] Received task: awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce] | |
[2018-02-14 14:40:31,820: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.handle_ha_toplogy_changes', 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', {'origin': 'gen210@awx', 'lang': 'py', 'task': 'awx.main.tasks.handle_ha_toplogy_changes', 'group': None, 'root_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u'tower_broadcast_all'}, 'expires': None, u'correlation_id': 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': '13957fad-e323-47fa-993a-2313a6463bd0', u'reply_to': '59e307f4-0247-3cdc-a845-d5bc61c06e4d', 'id': 'fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:31,829: INFO/ForkPoolWorker-5] Task awx.main.tasks.cluster_node_heartbeat[13957fad-e323-47fa-993a-2313a6463bd0] succeeded in 0.031104312s: None | |
[2018-02-14 14:40:31,833: DEBUG/MainProcess] Task accepted: awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce] pid:210 | |
2018-02-14 14:40:31,852 DEBUG awx.main.tasks Reconfigure celeryd queues task on host celery@awx | |
[2018-02-14 14:40:31,881: DEBUG/ForkPoolWorker-5] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:40:31,885: DEBUG/ForkPoolWorker-5] using channel_id: 1 | |
[2018-02-14 14:40:31,888: DEBUG/ForkPoolWorker-5] Channel open | |
[2018-02-14 14:40:31,892: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'90aadbb1-f68f-3e58-9267-63708d9df664', u'exchange': u'reply.celery.pidbox'} ticket:fdfaca9a-ec0f-4259-be61-0ae8b2dacac5] | |
[2018-02-14 14:40:31,906: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@29b69c25831f', 'platform': 'Erlang/OTP 20.1.7', 'version': '3.7.3'}, mechanisms: ['AMQPLAIN', 'PLAIN'], locales: [u'en_US'] | |
[2018-02-14 14:40:31,909: DEBUG/MainProcess] using channel_id: 1 | |
[2018-02-14 14:40:31,913: DEBUG/MainProcess] Channel open | |
2018-02-14 14:40:31,921 INFO awx.main.tasks Workers on tower node 'awx' removed from queues [] and added to queues [] | |
2018-02-14 14:40:31,925 INFO awx.main.tasks Worker on tower node 'awx' updated celery routes {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} all routes are now {'awx.main.tasks.purge_old_stdout_files': {'queue': u'awx', 'routing_key': u'awx'}, 'awx.main.tasks.cluster_node_heartbeat': {'queue': u'awx', 'routing_key': u'awx'}} | |
[2018-02-14 14:40:31,930: INFO/ForkPoolWorker-5] Task awx.main.tasks.handle_ha_toplogy_changes[fd9a7b0e-6736-4432-a0b1-b4410fe5c7ce] succeeded in 0.0928196149998s: None | |
[2018-02-14 14:40:41,081: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:41,086: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 196/557, now - 202/604, monotonic - 5273.08450469, last_heartbeat_sent - 5273.08447044, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:40:41,490: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:40:41,495: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] expires:[2018-02-14 14:41:01.493104+00:00] | |
[2018-02-14 14:40:41,496: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '088a9957-5a4c-44e7-a343-c7a857934b53', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '088a9957-5a4c-44e7-a343-c7a857934b53', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:01.493104+00:00', u'correlation_id': '088a9957-5a4c-44e7-a343-c7a857934b53', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '088a9957-5a4c-44e7-a343-c7a857934b53', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:40:41,499: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->088a9957-5a4c-44e7-a343-c7a857934b53 | |
2018-02-14 14:40:41,510 DEBUG awx.main.scheduler Running Tower task manager. | |
[2018-02-14 14:40:41,513: DEBUG/Beat] beat: Waking up in 20.04 seconds. | |
[2018-02-14 14:40:41,514: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] pid:443 | |
2018-02-14 14:40:41,524 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:40:41,551: INFO/ForkPoolWorker-6] Task awx.main.scheduler.tasks.run_task_manager[088a9957-5a4c-44e7-a343-c7a857934b53] succeeded in 0.0511622899994s: None | |
[2018-02-14 14:41:01,086: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:41:01,091: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 202/604, now - 203/637, monotonic - 5293.09034378, last_heartbeat_sent - 5293.09031439, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:41:01,574: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler) | |
[2018-02-14 14:41:01,592: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->d11a467b-be71-4c17-b682-2969859b4dc1 | |
[2018-02-14 14:41:01,599: INFO/MainProcess] Received task: awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] expires:[2018-02-14 14:41:21.580943+00:00] | |
[2018-02-14 14:41:01,600: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:41:01,601: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.tasks.awx_periodic_scheduler', 'd11a467b-be71-4c17-b682-2969859b4dc1', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.tasks.awx_periodic_scheduler', 'group': None, 'root_id': 'd11a467b-be71-4c17-b682-2969859b4dc1', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.580943+00:00', u'correlation_id': 'd11a467b-be71-4c17-b682-2969859b4dc1', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'd11a467b-be71-4c17-b682-2969859b4dc1', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:41:01,617: DEBUG/MainProcess] Task accepted: awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] pid:443 | |
2018-02-14 14:41:01,623 DEBUG awx.main.tasks Last scheduler run was: 2018-02-14 14:40:31.601264+00:00 | |
[2018-02-14 14:41:01,626: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] expires:[2018-02-14 14:41:21.606234+00:00] | |
[2018-02-14 14:41:01,627: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:21.606234+00:00', u'correlation_id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': 'dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:41:01,626: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865 | |
[2018-02-14 14:41:01,627: DEBUG/Beat] beat: Waking up in 19.96 seconds. | |
[2018-02-14 14:41:01,632: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] pid:210 | |
2018-02-14 14:41:01,633 DEBUG awx.main.scheduler Running Tower task manager. | |
2018-02-14 14:41:01,638 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:41:01,651: INFO/ForkPoolWorker-6] Task awx.main.tasks.awx_periodic_scheduler[d11a467b-be71-4c17-b682-2969859b4dc1] succeeded in 0.045172829s: None | |
[2018-02-14 14:41:01,665: INFO/ForkPoolWorker-5] Task awx.main.scheduler.tasks.run_task_manager[dc6a8ae1-39c2-4a2e-a7a2-2fbe17aa7865] succeeded in 0.037361273s: None | |
[2018-02-14 14:41:11,076: INFO/MainProcess] Scaling down 2 processes. | |
[2018-02-14 14:41:21,092: DEBUG/MainProcess] heartbeat_tick : for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:41:21,098: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 203/637, now - 205/673, monotonic - 5313.09638642, last_heartbeat_sent - 5313.09635675, heartbeat int. - 60 for connection a66f1e832d1c4ad681a0fcae6d65e774 | |
[2018-02-14 14:41:21,610: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager) | |
[2018-02-14 14:41:21,627: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->4668edd8-b5b6-4aec-af2a-6f458760ec45 | |
[2018-02-14 14:41:21,625: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] expires:[2018-02-14 14:41:41.615373+00:00] | |
[2018-02-14 14:41:21,634: DEBUG/Beat] beat: Waking up in 9.95 seconds. | |
[2018-02-14 14:41:21,635: INFO/MainProcess] Scaling up 1 processes. | |
[2018-02-14 14:41:21,713: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x37f4c80> (args:('awx.main.scheduler.tasks.run_task_manager', '4668edd8-b5b6-4aec-af2a-6f458760ec45', {'origin': 'gen205@awx', 'lang': 'py', 'task': 'awx.main.scheduler.tasks.run_task_manager', 'group': None, 'root_id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'tower', u'exchange': u''}, 'expires': '2018-02-14T14:41:41.615373+00:00', u'correlation_id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '()', 'eta': None, 'parent_id': None, u'reply_to': '00e331d2-05ae-3adb-81ba-d558f507de1e', 'id': '4668edd8-b5b6-4aec-af2a-6f458760ec45', 'kwargsrepr': '{}'}, u'[[], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{}) | |
[2018-02-14 14:41:21,717: DEBUG/MainProcess] basic.qos: prefetch_count->16 | |
[2018-02-14 14:41:21,724: DEBUG/ForkPoolWorker-7] Closed channel #1 | |
[2018-02-14 14:41:21,827: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] pid:444 | |
2018-02-14 14:41:21,836 DEBUG awx.main.scheduler Running Tower task manager. | |
2018-02-14 14:41:21,842 DEBUG awx.main.scheduler Starting Scheduler | |
[2018-02-14 14:41:21,877: INFO/ForkPoolWorker-7] Task awx.main.scheduler.tasks.run_task_manager[4668edd8-b5b6-4aec-af2a-6f458760ec45] succeeded in 0.0467407699998s: None |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment