Skip to content

Instantly share code, notes, and snippets.

@sunahsuh
Created December 20, 2016 16:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sunahsuh/885d5ad29348150a91715b405b57e989 to your computer and use it in GitHub Desktop.
Save sunahsuh/885d5ad29348150a91715b405b57e989 to your computer and use it in GitHub Desktop.
Zombie incident 12/18
[W 161218 00:34:27 state:74] Substantial drift from celery@255af4540669 may mean clocks are out of sync. Current drift is
17 seconds. [orig: 2016-12-18 00:34:27.938052 recv: 2016-12-18 00:34:10.375481]
[ec2-user@ip-10-10-1-173 ~]$ docker logs --since 2016-12-18 b3b36d067751
=INFO REPORT==== 18-Dec-2016::00:00:03 ===
accepting AMQP connection <0.28323.48> (172.17.0.5:33538 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:08 ===
closing AMQP connection <0.28323.48> (172.17.0.5:33538 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:08 ===
accepting AMQP connection <0.28427.48> (172.17.0.5:33670 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:00:08 ===
closing AMQP connection <0.28427.48> (172.17.0.5:33670 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:00:15 ===
accepting AMQP connection <0.28516.48> (172.17.0.5:33686 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:50 ===
closing AMQP connection <0.28516.48> (172.17.0.5:33686 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:50 ===
accepting AMQP connection <0.28821.48> (172.17.0.5:33716 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:50 ===
closing AMQP connection <0.28821.48> (172.17.0.5:33716 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:00:50 ===
accepting AMQP connection <0.28836.48> (172.17.0.5:33724 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:00:51 ===
closing AMQP connection <0.28836.48> (172.17.0.5:33724 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:00:57 ===
accepting AMQP connection <0.28906.48> (172.17.0.5:33740 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:01:32 ===
closing AMQP connection <0.28906.48> (172.17.0.5:33740 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:01:33 ===
accepting AMQP connection <0.29211.48> (172.17.0.5:33796 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:01:33 ===
closing AMQP connection <0.29211.48> (172.17.0.5:33796 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:01:33 ===
accepting AMQP connection <0.29245.48> (172.17.0.5:33804 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:01:33 ===
closing AMQP connection <0.29245.48> (172.17.0.5:33804 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:01:40 ===
accepting AMQP connection <0.29310.48> (172.17.0.5:33822 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:02:15 ===
closing AMQP connection <0.29310.48> (172.17.0.5:33822 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:02:16 ===
accepting AMQP connection <0.29620.48> (172.17.0.5:33892 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:02:46 ===
closing AMQP connection <0.29620.48> (172.17.0.5:33892 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:02:46 ===
accepting AMQP connection <0.29896.48> (172.17.0.5:33900 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:02:46 ===
closing AMQP connection <0.29896.48> (172.17.0.5:33900 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:31:34 ===
accepting AMQP connection <0.11018.49> (172.17.0.5:36142 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:32:09 ===
closing AMQP connection <0.11018.49> (172.17.0.5:36142 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:32:09 ===
accepting AMQP connection <0.11321.49> (172.17.0.5:36182 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:32:39 ===
closing AMQP connection <0.11321.49> (172.17.0.5:36182 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:32:40 ===
accepting AMQP connection <0.11622.49> (172.17.0.5:36204 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:32:40 ===
closing AMQP connection <0.11622.49> (172.17.0.5:36204 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:32:46 ===
accepting AMQP connection <0.11694.49> (172.17.0.5:36220 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:21 ===
closing AMQP connection <0.11694.49> (172.17.0.5:36220 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:22 ===
accepting AMQP connection <0.11996.49> (172.17.0.5:36246 -> 172.17.0.3:5672)
=ERROR REPORT==== 18-Dec-2016::00:33:22 ===
Error on AMQP connection <0.11996.49> (172.17.0.5:36246 -> 172.17.0.3:5672, vhost: 'airflow', user: 'airflow', state: closing), channel 1:
operation channel.close caused a connection exception channel_error: "expected 'channel.open'"
=INFO REPORT==== 18-Dec-2016::00:33:22 ===
closing AMQP connection <0.11996.49> (172.17.0.5:36246 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:22 ===
accepting AMQP connection <0.12041.49> (172.17.0.5:36254 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:33:22 ===
closing AMQP connection <0.12041.49> (172.17.0.5:36254 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:33:29 ===
accepting AMQP connection <0.12092.49> (172.17.0.5:36270 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:34 ===
closing AMQP connection <0.12092.49> (172.17.0.5:36270 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:34 ===
accepting AMQP connection <0.12165.49> (172.17.0.5:36284 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:39 ===
closing AMQP connection <0.12165.49> (172.17.0.5:36284 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:39 ===
accepting AMQP connection <0.12230.49> (172.17.0.5:36292 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:33:39 ===
closing AMQP connection <0.12230.49> (172.17.0.5:36292 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:33:46 ===
accepting AMQP connection <0.12282.49> (172.17.0.5:36308 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:51 ===
closing AMQP connection <0.12282.49> (172.17.0.5:36308 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:51 ===
accepting AMQP connection <0.12357.49> (172.17.0.5:36324 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:56 ===
closing AMQP connection <0.12357.49> (172.17.0.5:36324 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:33:56 ===
accepting AMQP connection <0.12406.49> (172.17.0.5:36332 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:33:56 ===
closing AMQP connection <0.12406.49> (172.17.0.5:36332 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:34:03 ===
accepting AMQP connection <0.12466.49> (172.17.0.5:36348 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:34:08 ===
closing AMQP connection <0.12466.49> (172.17.0.5:36348 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:34:08 ===
accepting AMQP connection <0.12531.49> (172.17.0.5:36362 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:34:43 ===
closing AMQP connection <0.12531.49> (172.17.0.5:36362 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:34:43 ===
accepting AMQP connection <0.12840.49> (172.17.0.5:36380 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:34:43 ===
closing AMQP connection <0.12840.49> (172.17.0.5:36380 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::00:34:50 ===
accepting AMQP connection <0.12899.49> (172.17.0.5:36396 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:35:25 ===
closing AMQP connection <0.12899.49> (172.17.0.5:36396 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:35:26 ===
accepting AMQP connection <0.13221.49> (172.17.0.5:36452 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:35:26 ===
closing AMQP connection <0.13221.49> (172.17.0.5:36452 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::00:35:26 ===
accepting AMQP connection <0.13229.49> (172.17.0.5:36460 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::00:35:26 ===
closing AMQP connection <0.13229.49> (172.17.0.5:36460 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:00:11 ===
accepting AMQP connection <0.25201.49> (172.17.0.5:38384 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:00:45 ===
closing AMQP connection <0.25201.49> (172.17.0.5:38384 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:00:45 ===
accepting AMQP connection <0.25498.49> (172.17.0.5:38414 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:00:45 ===
closing AMQP connection <0.25498.49> (172.17.0.5:38414 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:00:46 ===
accepting AMQP connection <0.25518.49> (172.17.0.5:38428 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:00:50 ===
closing AMQP connection <0.25518.49> (172.17.0.5:38428 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:03:21 ===
accepting AMQP connection <0.26778.49> (172.17.0.5:38632 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:03:26 ===
closing AMQP connection <0.26778.49> (172.17.0.5:38632 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:03:54 ===
accepting AMQP connection <0.27068.49> (172.17.0.5:38702 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:03:58 ===
closing AMQP connection <0.27068.49> (172.17.0.5:38702 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:03:58 ===
accepting AMQP connection <0.27116.49> (172.17.0.5:38730 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:04:33 ===
closing AMQP connection <0.27116.49> (172.17.0.5:38730 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:04:33 ===
accepting AMQP connection <0.27426.49> (172.17.0.5:38744 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:04:33 ===
closing AMQP connection <0.27426.49> (172.17.0.5:38744 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:04:45 ===
accepting AMQP connection <0.27525.49> (172.17.0.5:38766 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:04:50 ===
closing AMQP connection <0.27525.49> (172.17.0.5:38766 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:04:50 ===
accepting AMQP connection <0.27584.49> (172.17.0.5:38794 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:04:55 ===
closing AMQP connection <0.27584.49> (172.17.0.5:38794 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:05:27 ===
accepting AMQP connection <0.27884.49> (172.17.0.5:38844 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:06:02 ===
closing AMQP connection <0.27884.49> (172.17.0.5:38844 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:06:02 ===
accepting AMQP connection <0.28213.49> (172.17.0.5:38888 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:06:32 ===
closing AMQP connection <0.28213.49> (172.17.0.5:38888 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:06:33 ===
accepting AMQP connection <0.28470.49> (172.17.0.5:38896 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:06:33 ===
closing AMQP connection <0.28470.49> (172.17.0.5:38896 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:20:27 ===
accepting AMQP connection <0.2446.50> (172.17.0.5:39998 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:20:31 ===
closing AMQP connection <0.2446.50> (172.17.0.5:39998 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:20:31 ===
accepting AMQP connection <0.2491.50> (172.17.0.5:40032 -> 172.17.0.3:5672)
=ERROR REPORT==== 18-Dec-2016::01:20:36 ===
Error on AMQP connection <0.2491.50> (172.17.0.5:40032 -> 172.17.0.3:5672, vhost: 'airflow', user: 'airflow', state: closing), channel 1:
operation channel.close caused a connection exception channel_error: "expected 'channel.open'"
=INFO REPORT==== 18-Dec-2016::01:21:06 ===
closing AMQP connection <0.2491.50> (172.17.0.5:40032 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:21:06 ===
accepting AMQP connection <0.2797.50> (172.17.0.5:40046 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:21:06 ===
closing AMQP connection <0.2797.50> (172.17.0.5:40046 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::01:48:27 ===
accepting AMQP connection <0.16025.50> (172.17.0.5:42202 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:49:01 ===
closing AMQP connection <0.16025.50> (172.17.0.5:42202 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:49:01 ===
accepting AMQP connection <0.16310.50> (172.17.0.5:42236 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:49:01 ===
closing AMQP connection <0.16310.50> (172.17.0.5:42236 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::01:49:02 ===
accepting AMQP connection <0.16318.50> (172.17.0.5:42250 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::01:49:06 ===
closing AMQP connection <0.16318.50> (172.17.0.5:42250 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 18-Dec-2016::03:41:48 ===
accepting AMQP connection <0.5198.52> (172.17.0.5:50876 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::03:41:52 ===
closing AMQP connection <0.5198.52> (172.17.0.5:50876 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::03:41:52 ===
accepting AMQP connection <0.5261.52> (172.17.0.5:50946 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::03:41:57 ===
closing AMQP connection <0.5261.52> (172.17.0.5:50946 -> 172.17.0.3:5672)
=INFO REPORT==== 18-Dec-2016::03:41:57 ===
accepting AMQP connection <0.5326.52> (172.17.0.5:50960 -> 172.17.0.3:5672)
=WARNING REPORT==== 18-Dec-2016::03:42:02 ===
closing AMQP connection <0.5326.52> (172.17.0.5:50960 -> 172.17.0.3:5672):
client unexpectedly closed TCP connection
This file has been truncated, but you can view the full file.
[2016-12-18 00:00:03,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:03,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:03,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,134] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,136] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,139] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,144] {base_executor.py:36} INFO - Adding to queue: airflow run bugzilla_dataset update_bugs 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/bugzilla_dataset.py
[2016-12-18 00:00:03,148] {base_executor.py:36} INFO - Adding to queue: airflow run telemetry_aggregates telemetry_aggregate_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/telemetry_aggregates.py
[2016-12-18 00:00:03,204] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,211] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,214] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,215] {base_executor.py:36} INFO - Adding to queue: airflow run socorro_import socorro_import 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/socorro_import.py
[2016-12-18 00:00:03,217] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,222] {base_executor.py:36} INFO - Adding to queue: airflow run mobile_clients mobile_clients 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/mobile_clients.py
[2016-12-18 00:00:03,269] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,271] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,273] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,274] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,275] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,279] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,284] {base_executor.py:36} INFO - Adding to queue: airflow run client_count client_count_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/client_count.py
[2016-12-18 00:00:03,316] {base_executor.py:36} INFO - Adding to queue: airflow run longitudinal longitudinal 2016-12-11T00:00:00 --local -sd DAGS_FOLDER/longitudinal.py
[2016-12-18 00:00:03,339] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,341] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,344] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,349] {base_executor.py:36} INFO - Adding to queue: airflow run android_addons android_addons 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_addons.py
[2016-12-18 00:00:03,394] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,396] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,399] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,405] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,417] {base_executor.py:36} INFO - Adding to queue: airflow run sync_view sync_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/sync_view.py
[2016-12-18 00:00:03,439] {base_executor.py:36} INFO - Adding to queue: airflow run main_summary delayed_start 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/main_summary.py
[2016-12-18 00:00:03,470] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,472] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,476] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,481] {base_executor.py:36} INFO - Adding to queue: airflow run android_events android_events 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_events.py
[2016-12-18 00:00:03,496] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,498] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,503] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,508] {base_executor.py:36} INFO - Adding to queue: airflow run android_clients android_clients 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_clients.py
[2016-12-18 00:00:03,564] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:03,566] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:03,570] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:03,575] {base_executor.py:36} INFO - Adding to queue: airflow run crash_aggregates crash_aggregate_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/crash_aggregates.py
[2016-12-18 00:00:03,597] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:03,597] {jobs.py:744} INFO - Loop took: 0.585104 seconds
[2016-12-18 00:00:03,607] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:03,607] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:57:48.607483
[2016-12-18 00:00:03,619] {celery_executor.py:64} INFO - [celery] queuing ('longitudinal', 'longitudinal', datetime.datetime(2016, 12, 11, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,643] {celery_executor.py:64} INFO - [celery] queuing ('main_summary', 'delayed_start', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,650] {celery_executor.py:64} INFO - [celery] queuing ('sync_view', 'sync_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,655] {celery_executor.py:64} INFO - [celery] queuing ('telemetry_aggregates', 'telemetry_aggregate_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,661] {celery_executor.py:64} INFO - [celery] queuing ('mobile_clients', 'mobile_clients', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,670] {celery_executor.py:64} INFO - [celery] queuing ('bugzilla_dataset', 'update_bugs', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,677] {celery_executor.py:64} INFO - [celery] queuing ('android_addons', 'android_addons', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,686] {celery_executor.py:64} INFO - [celery] queuing ('android_events', 'android_events', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,696] {celery_executor.py:64} INFO - [celery] queuing ('client_count', 'client_count_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,706] {celery_executor.py:64} INFO - [celery] queuing ('android_clients', 'android_clients', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,714] {celery_executor.py:64} INFO - [celery] queuing ('crash_aggregates', 'crash_aggregate_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:03,724] {celery_executor.py:64} INFO - [celery] queuing ('socorro_import', 'socorro_import', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:08,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:08,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:08,130] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,130] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,180] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,181] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,181] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,183] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,185] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:08,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,232] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,235] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,237] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,237] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,240] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,281] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,283] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,286] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,337] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,339] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,340] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,342] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,345] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,386] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,388] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,391] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:08,421] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,423] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,426] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:08,477] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:08,478] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:08,481] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:08,506] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:08,507] {jobs.py:744} INFO - Loop took: 0.493606 seconds
[2016-12-18 00:00:08,517] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:08,517] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:57:53.517392
[2016-12-18 00:00:08,524] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:00:08,526] {jobs.py:759} ERROR - Tachycardia!
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:00:09,878] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:00:09,942] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:00:09,959] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:00:10,153] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:00:10,153] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:00:10,292] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:00:10,316] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:10,327] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:10,507] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:10,507] {jobs.py:744} INFO - Loop took: 0.197362 seconds
[2016-12-18 00:00:10,518] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:10,518] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:57:55.518309
[2016-12-18 00:00:15,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:15,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:15,099] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,100] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,102] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,105] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:15,156] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,158] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,159] {base_executor.py:36} INFO - Adding to queue: airflow run socorro_import socorro_import 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/socorro_import.py
[2016-12-18 00:00:15,161] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,203] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,208] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,211] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,216] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,264] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,266] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,269] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,319] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,325] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,331] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,333] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,336] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,372] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,374] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:15,383] {base_executor.py:36} INFO - Adding to queue: airflow run android_events android_events 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_events.py
[2016-12-18 00:00:15,427] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,429] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,433] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:15,485] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:15,487] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:15,491] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:15,497] {base_executor.py:36} INFO - Adding to queue: airflow run crash_aggregates crash_aggregate_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/crash_aggregates.py
[2016-12-18 00:00:15,522] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:15,522] {jobs.py:744} INFO - Loop took: 0.509889 seconds
[2016-12-18 00:00:15,532] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:15,533] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:00.533165
[2016-12-18 00:00:15,546] {celery_executor.py:64} INFO - [celery] queuing ('crash_aggregates', 'crash_aggregate_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:15,571] {celery_executor.py:64} INFO - [celery] queuing ('android_events', 'android_events', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:15,577] {celery_executor.py:64} INFO - [celery] queuing ('socorro_import', 'socorro_import', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:20,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:20,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:50,141] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,142] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,195] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,198] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:50,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,203] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,247] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,249] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,249] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,252] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,253] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,299] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,303] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,348] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,350] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,353] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,359] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,361] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,364] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,405] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:50,443] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,445] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,448] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:50,494] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:50,495] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:50,498] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:50,524] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:50,524] {jobs.py:744} INFO - Loop took: 30.51251 seconds
[2016-12-18 00:00:50,535] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:50,535] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:35.535339
[2016-12-18 00:00:50,542] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:00:50,543] {jobs.py:759} ERROR - Tachycardia!
[2016-12-18 00:00:50,549] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:50,558] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:50,738] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:50,738] {jobs.py:744} INFO - Loop took: 0.194528 seconds
[2016-12-18 00:00:50,748] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:50,749] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:35.749138
[2016-12-18 00:00:50,780] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:50,789] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:50,969] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:50,969] {jobs.py:744} INFO - Loop took: 0.194565 seconds
[2016-12-18 00:00:50,979] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:50,980] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:35.980127
[2016-12-18 00:00:50,987] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:00:50,987] {jobs.py:759} ERROR - Tachycardia!
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:00:52,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:00:52,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:00:52,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:00:52,584] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:00:52,585] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:00:52,717] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:00:52,740] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:52,751] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:52,927] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:52,928] {jobs.py:744} INFO - Loop took: 0.193715 seconds
[2016-12-18 00:00:52,938] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:52,938] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:37.938524
[2016-12-18 00:00:57,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:00:57,024] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:00:57,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,091] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,143] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:57,155] {base_executor.py:36} INFO - Adding to queue: airflow run socorro_import socorro_import 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/socorro_import.py
[2016-12-18 00:00:57,191] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,200] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,246] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,248] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,251] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,293] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,295] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,297] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,307] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,309] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,312] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,338] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,340] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,343] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:57,348] {base_executor.py:36} INFO - Adding to queue: airflow run android_events android_events 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_events.py
[2016-12-18 00:00:57,389] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,391] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,394] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:00:57,436] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:00:57,438] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:00:57,441] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:00:57,445] {base_executor.py:36} INFO - Adding to queue: airflow run crash_aggregates crash_aggregate_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/crash_aggregates.py
[2016-12-18 00:00:57,465] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:00:57,465] {jobs.py:744} INFO - Loop took: 0.455208 seconds
[2016-12-18 00:00:57,475] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:00:57,476] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:58:42.475989
[2016-12-18 00:00:57,488] {celery_executor.py:64} INFO - [celery] queuing ('crash_aggregates', 'crash_aggregate_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:57,513] {celery_executor.py:64} INFO - [celery] queuing ('android_events', 'android_events', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:00:57,519] {celery_executor.py:64} INFO - [celery] queuing ('socorro_import', 'socorro_import', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:01:02,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:02,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:01:32,099] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,110] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,112] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,115] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,165] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,167] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,169] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,198] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,202] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,213] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,215] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,218] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,270] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,272] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,275] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,303] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,305] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,308] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,318] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,323] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,374] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,375] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,378] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:32,384] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,386] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,389] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:32,431] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:32,432] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:32,435] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:32,460] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:01:32,460] {jobs.py:744} INFO - Loop took: 30.448131 seconds
[2016-12-18 00:01:32,477] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:01:32,477] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:59:17.477739
[2016-12-18 00:01:32,483] {models.py:1286} ERROR - <TaskInstance: longitudinal.longitudinal 2016-12-11 00:00:00 [running]> killed as zombie
None
[2016-12-18 00:01:32,483] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2016-12-18 00:01:32,637] {email.py:96} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'rvitillo@mozilla.com', 'rharter@mozilla.com']
[2016-12-18 00:01:32,804] {models.py:1327} ERROR - <TaskInstance: longitudinal.longitudinal 2016-12-11 00:00:00 [running]> killed as zombie
[2016-12-18 00:01:32,807] {models.py:333} INFO - Marked zombie job <TaskInstance: longitudinal.longitudinal 2016-12-11 00:00:00 [up_for_retry]> as failed
[2016-12-18 00:01:32,809] {models.py:1286} ERROR - <TaskInstance: bugzilla_dataset.update_bugs 2016-12-17 00:00:00 [running]> killed as zombie
None
[2016-12-18 00:01:32,809] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2016-12-18 00:01:32,923] {email.py:96} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'mdoglio@mozilla.com']
[2016-12-18 00:01:33,091] {models.py:1327} ERROR - <TaskInstance: bugzilla_dataset.update_bugs 2016-12-17 00:00:00 [running]> killed as zombie
[2016-12-18 00:01:33,094] {models.py:333} INFO - Marked zombie job <TaskInstance: bugzilla_dataset.update_bugs 2016-12-17 00:00:00 [up_for_retry]> as failed
[2016-12-18 00:01:33,097] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:01:33,098] {jobs.py:759} ERROR - Tachycardia!
[2016-12-18 00:01:33,103] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:33,112] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:01:33,288] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:01:33,288] {jobs.py:744} INFO - Loop took: 0.189777 seconds
[2016-12-18 00:01:33,298] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:01:33,298] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:59:18.298726
[2016-12-18 00:01:33,336] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:33,345] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:01:33,523] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:01:33,524] {jobs.py:744} INFO - Loop took: 0.192516 seconds
[2016-12-18 00:01:33,534] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:01:33,534] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:59:18.534429
[2016-12-18 00:01:33,541] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:01:33,542] {jobs.py:759} ERROR - Tachycardia!
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:01:34,892] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:01:34,960] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:01:34,979] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:01:35,180] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:01:35,180] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:01:35,320] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:01:35,344] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:35,354] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:01:35,544] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:01:35,544] {jobs.py:744} INFO - Loop took: 0.206437 seconds
[2016-12-18 00:01:35,554] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:01:35,554] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:59:20.554924
[2016-12-18 00:01:40,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:40,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:01:40,093] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:40,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,147] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,152] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,196] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,201] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:40,264] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,266] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,269] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,311] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,313] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,313] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,315] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,316] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,318] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,362] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:40,372] {base_executor.py:36} INFO - Adding to queue: airflow run android_events android_events 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/android_events.py
[2016-12-18 00:01:40,402] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,404] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,407] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:01:40,451] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:01:40,453] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:01:40,456] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:01:40,461] {base_executor.py:36} INFO - Adding to queue: airflow run crash_aggregates crash_aggregate_view 2016-12-17T00:00:00 --local -sd DAGS_FOLDER/crash_aggregates.py
[2016-12-18 00:01:40,486] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:01:40,486] {jobs.py:744} INFO - Loop took: 0.474443 seconds
[2016-12-18 00:01:40,496] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:01:40,496] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-17 23:59:25.496411
[2016-12-18 00:01:40,501] {models.py:1286} ERROR - <TaskInstance: telemetry_aggregates.telemetry_aggregate_view 2016-12-17 00:00:00 [running]> killed as zombie
None
[2016-12-18 00:01:40,501] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2016-12-18 00:01:40,623] {email.py:96} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'rvitillo@mozilla.com', 'fbertsch@mozilla.com']
[2016-12-18 00:01:40,769] {models.py:1327} ERROR - <TaskInstance: telemetry_aggregates.telemetry_aggregate_view 2016-12-17 00:00:00 [running]> killed as zombie
[2016-12-18 00:01:40,772] {models.py:333} INFO - Marked zombie job <TaskInstance: telemetry_aggregates.telemetry_aggregate_view 2016-12-17 00:00:00 [up_for_retry]> as failed
[2016-12-18 00:01:40,779] {celery_executor.py:64} INFO - [celery] queuing ('crash_aggregates', 'crash_aggregate_view', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:01:40,804] {celery_executor.py:64} INFO - [celery] queuing ('android_events', 'android_events', datetime.datetime(2016, 12, 17, 0, 0)) through celery, queue=default
[2016-12-18 00:01:45,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:01:45,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:15,111] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,113] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,116] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:15,120] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,122] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,125] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:15,182] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,183] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,184] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,187] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,238] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,241] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,243] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,244] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:15,246] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,250] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,298] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,303] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,353] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,356] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,370] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,372] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,375] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,406] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,412] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,460] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,465] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,509] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:15,510] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:15,513] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:15,534] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:15,534] {jobs.py:744} INFO - Loop took: 30.523271 seconds
[2016-12-18 00:02:15,544] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:15,544] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:00.544832
[2016-12-18 00:02:15,549] {models.py:1286} ERROR - <TaskInstance: crash_aggregates.crash_aggregate_view 2016-12-17 00:00:00 [running]> killed as zombie
None
[2016-12-18 00:02:15,550] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2016-12-18 00:02:15,673] {email.py:96} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'mdoglio@mozilla.com']
[2016-12-18 00:02:15,827] {models.py:1327} ERROR - <TaskInstance: crash_aggregates.crash_aggregate_view 2016-12-17 00:00:00 [running]> killed as zombie
[2016-12-18 00:02:15,830] {models.py:333} INFO - Marked zombie job <TaskInstance: crash_aggregates.crash_aggregate_view 2016-12-17 00:00:00 [up_for_retry]> as failed
[2016-12-18 00:02:15,832] {models.py:1286} ERROR - <TaskInstance: sync_view.sync_view 2016-12-17 00:00:00 [running]> killed as zombie
None
[2016-12-18 00:02:15,832] {models.py:1298} INFO - Marking task as UP_FOR_RETRY
[2016-12-18 00:02:15,958] {email.py:96} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'markh@mozilla.com', 'tchiovoloni@mozilla.com']
[2016-12-18 00:02:16,112] {models.py:1327} ERROR - <TaskInstance: sync_view.sync_view 2016-12-17 00:00:00 [running]> killed as zombie
[2016-12-18 00:02:16,115] {models.py:333} INFO - Marked zombie job <TaskInstance: sync_view.sync_view 2016-12-17 00:00:00 [up_for_retry]> as failed
[2016-12-18 00:02:16,118] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:02:16,119] {jobs.py:759} ERROR - Tachycardia!
[2016-12-18 00:02:16,124] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:02:16,133] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:16,306] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:16,307] {jobs.py:744} INFO - Loop took: 0.187663 seconds
[2016-12-18 00:02:16,316] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:16,317] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:01.317119
[2016-12-18 00:02:16,347] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:02:16,355] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:46,455] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,457] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,460] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:46,465] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,467] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,470] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:46,508] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,509] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,512] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,527] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,529] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,531] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,554] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,556] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,559] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:46,574] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,576] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,578] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,620] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,622] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,624] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,662] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,664] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,667] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,667] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,669] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,672] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:46,720] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,722] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,725] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,742] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,744] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,747] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:46,790] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:46,792] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:46,794] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:46,826] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:46,826] {jobs.py:744} INFO - Loop took: 30.484978 seconds
[2016-12-18 00:02:46,836] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:46,836] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:31.836713
[2016-12-18 00:02:46,843] {jobs.py:758} ERROR - [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 755, in _execute
executor.heartbeat()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/base_executor.py", line 107, in heartbeat
self.sync()
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 74, in sync
state = async.state
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 394, in state
return self._get_task_meta()['status']
File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/usr/local/lib/python2.7/dist-packages/celery/backends/amqp.py", line 163, in get_task_meta
binding.declare()
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 521, in declare
self.exchange.declare(nowait)
File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
nowait=nowait, passive=passive,
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
self._send_method((40, 10), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2016-12-18 00:02:46,844] {jobs.py:759} ERROR - Tachycardia!
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:02:48,163] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:02:48,225] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:02:48,243] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:02:48,426] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:02:48,426] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:02:48,553] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:02:48,576] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:02:48,586] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:48,774] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:48,774] {jobs.py:744} INFO - Loop took: 0.204285 seconds
[2016-12-18 00:02:48,784] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:48,785] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:33.785058
[2016-12-18 00:02:53,025] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:02:53,033] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:53,085] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,087] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,090] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:53,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:53,138] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,140] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,189] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,189] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,193] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:53,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,237] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,238] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,241] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,283] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,285] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,288] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:53,297] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,302] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,344] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,346] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,349] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,377] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,379] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,382] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:53,425] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:53,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:53,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:53,458] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:53,458] {jobs.py:744} INFO - Loop took: 0.438867 seconds
[2016-12-18 00:02:53,468] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:53,468] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:38.468571
[2016-12-18 00:02:58,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:02:58,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:02:58,074] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,076] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,078] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:58,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,083] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:58,127] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,129] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,175] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,180] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:58,183] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,229] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,231] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,234] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,276] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,278] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,280] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:58,289] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,294] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,328] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,330] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,333] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,375] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,377] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,380] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:02:58,431] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:02:58,433] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:02:58,436] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:02:58,464] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:02:58,464] {jobs.py:744} INFO - Loop took: 0.451532 seconds
[2016-12-18 00:02:58,474] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:02:58,474] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:43.474711
[2016-12-18 00:03:03,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:03,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:03,079] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,079] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:03,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:03,140] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,145] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,195] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,198] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,203] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:03,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,300] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,305] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:03,319] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,325] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,356] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,406] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,412] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:03,460] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:03,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:03,465] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:03,493] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:03,493] {jobs.py:744} INFO - Loop took: 0.481682 seconds
[2016-12-18 00:03:03,504] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:03,504] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:48.504345
[2016-12-18 00:03:08,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:08,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:08,099] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,102] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:08,104] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,107] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:08,155] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,158] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,205] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,207] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,207] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,210] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:08,212] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,256] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,312] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,314] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,317] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:08,318] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,323] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,368] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,370] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,374] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,404] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,406] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,409] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:08,457] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:08,459] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:08,462] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:08,490] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:08,490] {jobs.py:744} INFO - Loop took: 0.478441 seconds
[2016-12-18 00:03:08,500] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:08,500] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:00:53.500470
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:03:14,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:03:14,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:03:14,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:03:14,590] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:03:14,590] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:03:14,717] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:03:14,740] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:14,749] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:14,788] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,789] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,790] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,791] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,793] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:14,794] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:14,841] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,843] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,843] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,844] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,845] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:14,847] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:14,888] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,890] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,890] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,892] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,892] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:14,894] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:14,940] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,942] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,944] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:14,986] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,988] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,991] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:14,995] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:14,996] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:14,999] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:15,048] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:15,050] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:15,053] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:15,081] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:15,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:15,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:15,127] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:15,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:15,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:15,161] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:15,161] {jobs.py:744} INFO - Loop took: 0.42694 seconds
[2016-12-18 00:03:15,171] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:15,171] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:00.171269
[2016-12-18 00:03:19,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:19,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:19,079] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,083] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:19,085] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,088] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:19,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,143] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:19,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:19,186] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,188] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,190] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:19,191] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:19,237] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,239] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,242] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:19,284] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,286] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,289] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:19,292] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:19,294] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:19,297] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:19,398] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:19,398] {jobs.py:744} INFO - Loop took: 0.385776 seconds
[2016-12-18 00:03:19,408] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:19,408] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:04.408810
[2016-12-18 00:03:24,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:24,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:24,128] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:24,136] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,138] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,141] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:24,183] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,184] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,190] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,231] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,235] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,244] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,246] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,248] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:24,276] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,278] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,281] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,322] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,325] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:24,352] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,354] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,357] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,378] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,380] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,383] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,428] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,430] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,433] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:24,474] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:24,476] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:24,478] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:24,502] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:24,502] {jobs.py:744} INFO - Loop took: 0.490489 seconds
[2016-12-18 00:03:24,511] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:24,512] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:09.512081
[2016-12-18 00:03:29,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:29,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:29,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:29,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:29,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,145] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,194] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:29,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,240] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,242] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,245] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,289] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,294] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:29,300] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,305] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,341] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,343] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,345] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,380] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,382] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,384] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:29,428] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:29,430] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:29,433] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:29,457] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:29,457] {jobs.py:744} INFO - Loop took: 0.445445 seconds
[2016-12-18 00:03:29,467] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:29,467] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:14.467817
[2016-12-18 00:03:34,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:34,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:34,096] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,097] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:34,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:34,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,152] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,199] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,201] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:34,203] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,245] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,250] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,299] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,304] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:34,305] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,310] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,351] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,353] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,355] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,383] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,385] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,387] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:34,436] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:34,437] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:34,441] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:34,466] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:34,466] {jobs.py:744} INFO - Loop took: 0.454467 seconds
[2016-12-18 00:03:34,476] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:34,477] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:19.477001
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:03:40,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:03:40,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:03:40,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:03:40,606] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:03:40,607] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:03:40,736] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:03:40,760] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:40,770] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:40,808] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,810] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,812] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,813] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:40,814] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,817] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:40,861] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,863] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,864] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,866] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:40,866] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,869] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:40,907] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,909] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,911] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,912] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:40,912] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,915] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:40,958] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:40,960] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:40,963] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:41,004] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:41,006] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:41,009] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:41,014] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:41,016] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:41,019] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:41,066] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:41,068] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:41,070] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:41,101] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:41,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:41,106] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:41,155] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:41,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:41,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:41,189] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:41,189] {jobs.py:744} INFO - Loop took: 0.434686 seconds
[2016-12-18 00:03:41,200] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:41,200] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:26.200243
[2016-12-18 00:03:45,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:45,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:45,125] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,127] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,128] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:45,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:45,177] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,181] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:45,183] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:45,225] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,227] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,229] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:45,234] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,239] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:45,281] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,283] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,286] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:45,328] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:45,330] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:45,333] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:45,382] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:45,383] {jobs.py:744} INFO - Loop took: 0.369297 seconds
[2016-12-18 00:03:45,393] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:45,393] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:30.393523
[2016-12-18 00:03:50,023] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:50,032] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:50,096] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,097] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:50,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:50,155] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,156] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,158] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,203] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,207] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:50,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,263] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,265] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,268] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,318] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,323] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,323] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:50,325] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,329] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,375] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,377] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,380] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,418] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,420] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,423] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:50,481] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:50,483] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:50,486] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:50,514] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:50,514] {jobs.py:744} INFO - Loop took: 0.496606 seconds
[2016-12-18 00:03:50,524] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:50,525] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:35.525173
[2016-12-18 00:03:55,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:03:55,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:03:55,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:55,095] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:55,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,191] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,195] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,195] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:55,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,296] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,298] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,299] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,300] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:55,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,304] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,347] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,349] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,352] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,386] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,388] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,390] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:03:55,433] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:03:55,435] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:03:55,438] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:03:55,469] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:03:55,469] {jobs.py:744} INFO - Loop took: 0.4551 seconds
[2016-12-18 00:03:55,479] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:03:55,479] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:40.479684
[2016-12-18 00:04:00,025] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:00,037] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:00,105] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,107] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,110] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:00,112] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,114] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,116] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:00,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,172] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,174] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,176] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,206] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,208] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,211] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,219] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,221] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,224] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:00,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,254] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,257] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,306] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,310] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:00,328] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,329] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,332] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,358] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,360] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,362] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,417] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,419] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,422] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:00,464] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:00,466] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:00,469] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:00,500] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:00,500] {jobs.py:744} INFO - Loop took: 0.481323 seconds
[2016-12-18 00:04:00,510] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:00,510] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:45.510341
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:04:06,324] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:04:06,386] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:04:06,402] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:04:06,644] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:04:06,644] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:04:06,785] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:04:06,809] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:06,819] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:06,858] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,859] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,860] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,863] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:06,864] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:06,912] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,914] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,914] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,915] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,916] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:06,918] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:06,958] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,959] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:06,960] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,961] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:06,962] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:06,964] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:07,017] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,019] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,022] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:07,067] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,068] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,071] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:07,074] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,076] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,079] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:07,126] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:07,158] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:07,208] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:07,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:07,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:07,240] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:07,240] {jobs.py:744} INFO - Loop took: 0.437595 seconds
[2016-12-18 00:04:07,250] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:07,250] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:52.250342
[2016-12-18 00:04:11,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:11,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:11,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,095] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:11,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:11,147] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:11,154] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:11,194] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,199] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:11,208] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,213] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:11,243] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:11,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:11,248] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:11,381] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:11,382] {jobs.py:744} INFO - Loop took: 0.370671 seconds
[2016-12-18 00:04:11,397] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:11,397] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:01:56.397673
[2016-12-18 00:04:16,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:16,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:16,105] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,105] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,106] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,107] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,109] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:16,110] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:16,158] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,162] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,168] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,211] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,213] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,215] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,215] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:16,218] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,261] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,263] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,266] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,317] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,320] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,321] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:16,322] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,325] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,368] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,370] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,372] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,398] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,400] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,403] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:16,454] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:16,456] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:16,459] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:16,485] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:16,485] {jobs.py:744} INFO - Loop took: 0.473421 seconds
[2016-12-18 00:04:16,495] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:16,495] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:01.495327
[2016-12-18 00:04:21,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:21,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:21,087] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,091] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:21,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:21,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,151] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,191] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,196] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:21,198] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,203] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,245] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,250] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,293] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,295] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,297] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:21,302] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,304] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,307] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,345] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,347] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,350] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,383] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,385] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,388] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:21,430] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:21,432] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:21,434] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:21,459] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:21,459] {jobs.py:744} INFO - Loop took: 0.447408 seconds
[2016-12-18 00:04:21,469] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:21,469] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:06.469745
[2016-12-18 00:04:26,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:26,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:26,076] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,078] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,080] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:26,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:26,130] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,176] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,180] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:26,188] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,190] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,193] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,235] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,239] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,282] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,284] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,284] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,286] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,287] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,288] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:26,335] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,337] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,339] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,361] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,362] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,365] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:26,408] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:26,410] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:26,412] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:26,437] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:26,437] {jobs.py:744} INFO - Loop took: 0.424128 seconds
[2016-12-18 00:04:26,447] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:26,448] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:11.448042
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:04:32,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:04:32,383] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:04:32,400] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:04:32,642] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:04:32,642] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:04:32,977] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:04:33,001] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:33,011] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:33,054] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,055] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,056] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,057] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,058] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:33,059] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:33,109] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,110] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,113] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,113] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,115] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,117] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,156] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,159] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,160] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:33,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,164] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,208] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,254] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,256] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:33,264] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,266] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,269] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,308] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,310] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,313] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,342] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,344] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,347] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:33,394] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:33,396] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:33,400] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:33,427] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:33,428] {jobs.py:744} INFO - Loop took: 0.433177 seconds
[2016-12-18 00:04:33,438] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:33,438] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:18.438715
[2016-12-18 00:04:37,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:37,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:37,223] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:37,223] {jobs.py:744} INFO - Loop took: 0.209696 seconds
[2016-12-18 00:04:37,233] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:37,234] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:22.234026
[2016-12-18 00:04:42,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:42,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:42,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:42,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:42,188] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,190] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,190] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,193] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,236] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,238] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,238] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,240] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,241] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:42,243] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,293] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,294] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,297] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,340] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,345] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:42,346] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,347] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,350] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,402] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,404] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,407] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,426] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,428] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,431] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:42,477] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:42,479] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:42,482] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:42,506] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:42,506] {jobs.py:744} INFO - Loop took: 0.494348 seconds
[2016-12-18 00:04:42,516] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:42,516] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:27.516354
[2016-12-18 00:04:47,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:47,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:47,095] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,096] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:47,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:47,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,151] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,152] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,198] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,203] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:47,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,246] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,248] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,251] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,301] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,303] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,306] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,306] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:47,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,311] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,357] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,359] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,362] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,397] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,399] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,402] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:47,450] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:47,452] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:47,455] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:47,491] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:47,491] {jobs.py:744} INFO - Loop took: 0.480026 seconds
[2016-12-18 00:04:47,501] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:47,502] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:32.502065
[2016-12-18 00:04:52,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:52,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:52,077] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:52,082] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,084] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,088] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:52,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,143] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,199] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,202] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:52,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,250] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,252] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,255] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,300] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,305] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:52,315] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,318] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,320] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,356] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,412] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,414] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,417] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:52,465] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:52,467] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:52,470] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:52,498] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:52,498] {jobs.py:744} INFO - Loop took: 0.486462 seconds
[2016-12-18 00:04:52,508] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:52,508] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:37.508310
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:04:58,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:04:58,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:04:58,402] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:04:58,594] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:04:58,594] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:04:58,721] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:04:58,744] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:04:58,754] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:04:58,792] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,794] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,797] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,797] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:58,798] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,801] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:58,845] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,847] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,849] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,849] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:58,851] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,854] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:58,891] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,893] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,895] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:58,903] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,905] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,908] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:58,950] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,951] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,954] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:58,996] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,997] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:58,998] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:58,999] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:59,001] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:59,002] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:59,049] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:59,051] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:59,055] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:59,078] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:59,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:59,083] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:04:59,131] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:04:59,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:04:59,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:04:59,164] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:04:59,164] {jobs.py:744} INFO - Loop took: 0.426411 seconds
[2016-12-18 00:04:59,174] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:04:59,174] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:44.174882
[2016-12-18 00:05:03,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:03,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:03,075] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,079] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,080] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:03,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:03,131] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,135] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,136] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:03,137] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,139] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:03,180] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,183] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,186] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,186] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:03,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:03,232] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,237] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:03,287] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,292] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:03,293] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:03,295] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:03,298] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:03,405] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:03,405] {jobs.py:744} INFO - Loop took: 0.39403 seconds
[2016-12-18 00:05:03,416] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:03,416] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:48.416325
[2016-12-18 00:05:08,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:08,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:08,125] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,126] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,127] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:08,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:08,177] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,180] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,226] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,228] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,229] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,230] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:08,230] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,233] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,276] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,278] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,281] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,323] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,325] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,328] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:08,333] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,335] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,338] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,378] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,379] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,382] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,411] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,413] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,416] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:08,465] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:08,467] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:08,470] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:08,498] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:08,498] {jobs.py:744} INFO - Loop took: 0.486946 seconds
[2016-12-18 00:05:08,508] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:08,508] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:53.508370
[2016-12-18 00:05:13,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:13,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:13,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:13,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:13,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,196] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,196] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:13,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,242] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,244] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,246] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,289] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,294] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:13,298] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,303] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,340] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,344] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,376] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,378] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,380] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:13,426] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:13,428] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:13,431] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:13,455] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:13,455] {jobs.py:744} INFO - Loop took: 0.443475 seconds
[2016-12-18 00:05:13,465] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:13,466] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:02:58.466054
[2016-12-18 00:05:18,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:18,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:18,096] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,100] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:18,102] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,105] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:18,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,153] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,155] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,158] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,203] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,206] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:18,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,250] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,252] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,254] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,296] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,298] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,300] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:18,310] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,312] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,315] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,347] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,349] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,352] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,389] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,391] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,394] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:18,443] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:18,445] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:18,447] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:18,472] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:18,472] {jobs.py:744} INFO - Loop took: 0.460646 seconds
[2016-12-18 00:05:18,482] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:18,482] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:03.482437
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:05:24,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:05:24,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:05:24,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:05:24,640] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:05:24,640] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:05:24,767] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:05:24,790] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:24,800] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:24,838] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,840] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,842] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,843] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:24,844] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,847] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:24,890] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,892] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,895] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:24,895] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,896] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,899] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:24,936] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,938] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,940] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,940] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:24,942] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,944] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:24,986] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:24,988] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:24,990] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:25,039] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:25,041] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:25,044] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:25,057] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:25,059] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:25,063] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:25,098] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:25,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:25,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:25,146] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:25,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:25,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:25,199] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:25,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:25,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:25,232] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:25,232] {jobs.py:744} INFO - Loop took: 0.448049 seconds
[2016-12-18 00:05:25,244] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:25,244] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:10.244791
[2016-12-18 00:05:29,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:29,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:29,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,098] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:29,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:29,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:29,154] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:29,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,202] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,206] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:29,207] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:29,454] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:29,456] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:29,459] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:29,527] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:29,527] {jobs.py:744} INFO - Loop took: 0.516116 seconds
[2016-12-18 00:05:29,537] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:29,537] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:14.537924
[2016-12-18 00:05:34,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:34,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:34,074] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,076] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,078] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:34,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,083] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:34,126] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,130] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,136] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,172] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,174] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,177] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:34,177] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,231] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,233] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,236] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,278] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,279] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,281] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,283] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,284] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:34,338] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,340] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,343] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,355] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,359] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:34,401] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:34,403] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:34,406] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:34,429] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:34,429] {jobs.py:744} INFO - Loop took: 0.416021 seconds
[2016-12-18 00:05:34,440] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:34,440] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:19.440451
[2016-12-18 00:05:39,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:39,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:39,093] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:39,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:39,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,161] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,162] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,205] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,207] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,214] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,216] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,219] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:39,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,254] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,299] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,303] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:39,341] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,343] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,346] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,349] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,354] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,433] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,435] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,438] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:39,487] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:39,489] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:39,492] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:39,520] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:39,520] {jobs.py:744} INFO - Loop took: 0.508903 seconds
[2016-12-18 00:05:39,530] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:39,530] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:24.530740
[2016-12-18 00:05:44,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:44,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:44,076] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,077] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,078] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:44,082] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:44,130] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,132] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,134] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,181] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,183] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,183] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,186] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:44,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,231] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,233] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,235] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,287] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,292] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,292] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:44,293] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,296] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,340] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,345] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,372] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,374] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:44,423] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:44,425] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:44,428] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:44,451] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:44,451] {jobs.py:744} INFO - Loop took: 0.440271 seconds
[2016-12-18 00:05:44,461] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:44,462] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:29.462109
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:05:50,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:05:50,380] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:05:50,397] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:05:50,636] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:05:50,636] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:05:50,764] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:05:50,787] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:50,796] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:50,835] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,837] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,839] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,840] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:50,841] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,844] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:50,887] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,889] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,891] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:50,892] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,893] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,896] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:50,935] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,937] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,940] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:50,946] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,948] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,951] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:50,992] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:50,994] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:50,997] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:51,040] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:51,042] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:51,042] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:51,044] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:51,045] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:51,047] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:51,095] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:51,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:51,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:51,131] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:51,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:51,136] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:51,178] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:51,180] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:51,183] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:51,210] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:51,211] {jobs.py:744} INFO - Loop took: 0.430125 seconds
[2016-12-18 00:05:51,220] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:51,221] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:36.221152
[2016-12-18 00:05:55,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:05:55,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:05:55,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:55,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:55,190] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,192] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:55,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:55,241] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,243] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,245] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,246] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:55,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,250] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:05:55,297] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:05:55,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:05:55,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:05:55,423] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:05:55,424] {jobs.py:744} INFO - Loop took: 0.410682 seconds
[2016-12-18 00:05:55,434] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:05:55,434] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:40.434263
[2016-12-18 00:06:00,026] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:00,034] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:00,087] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:00,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:00,144] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,148] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,202] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,206] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:00,206] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,249] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,251] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,254] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,297] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:00,320] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,324] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,351] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,353] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,356] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,403] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,405] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,408] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:00,458] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:00,460] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:00,463] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:00,488] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:00,488] {jobs.py:744} INFO - Loop took: 0.469183 seconds
[2016-12-18 00:06:00,498] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:00,498] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:45.498614
[2016-12-18 00:06:05,025] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:05,034] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:05,105] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,105] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,107] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,107] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,110] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:05,110] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:05,171] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,173] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,173] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,175] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,176] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,179] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,221] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,222] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,225] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:05,226] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,228] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,231] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,279] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,281] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,284] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,328] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,329] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,333] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,333] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,335] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,338] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:05,391] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,393] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,396] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,407] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:05,455] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:05,457] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:05,459] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:05,483] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:05,483] {jobs.py:744} INFO - Loop took: 0.463947 seconds
[2016-12-18 00:06:05,493] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:05,493] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:50.493862
[2016-12-18 00:06:10,022] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:10,031] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:10,128] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,132] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:10,134] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:10,190] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,192] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,239] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,241] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,242] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,244] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,244] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:10,247] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,302] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,304] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,307] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,355] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:10,363] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,365] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,367] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,413] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,415] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,419] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,450] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,452] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,455] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:10,504] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:10,507] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:10,510] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:10,538] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:10,538] {jobs.py:744} INFO - Loop took: 0.522225 seconds
[2016-12-18 00:06:10,549] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:10,549] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:03:55.549258
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:06:16,322] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:06:16,384] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:06:16,401] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:06:16,593] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:06:16,593] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:06:16,721] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:06:16,744] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:16,754] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:16,794] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,796] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,799] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,799] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:16,800] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,803] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:16,847] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,849] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,851] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,852] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:16,853] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,856] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:16,900] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,902] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,902] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,903] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,905] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:16,906] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:16,949] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:16,951] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:16,953] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:17,003] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:17,005] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:17,008] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:17,008] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:17,010] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:17,014] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:17,058] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:17,060] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:17,062] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:17,094] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:17,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:17,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:17,144] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:17,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:17,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:17,173] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:17,173] {jobs.py:744} INFO - Loop took: 0.435116 seconds
[2016-12-18 00:06:17,183] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:17,183] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:02.183799
[2016-12-18 00:06:21,015] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:21,023] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:21,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:21,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:21,140] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,145] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:21,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:21,190] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,191] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,195] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:21,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:21,243] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,248] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:21,296] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,298] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:21,309] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:21,311] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:21,315] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:21,432] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:21,432] {jobs.py:744} INFO - Loop took: 0.422628 seconds
[2016-12-18 00:06:21,442] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:21,442] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:06.442633
[2016-12-18 00:06:26,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:26,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:26,123] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:26,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:26,174] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,176] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,179] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,179] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,181] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,185] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,228] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,230] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,230] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,233] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:26,235] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,282] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,284] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,287] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,333] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,335] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,338] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:26,349] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,354] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,388] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,390] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,393] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,442] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,444] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,447] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:26,495] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:26,497] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:26,500] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:26,528] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:26,528] {jobs.py:744} INFO - Loop took: 0.517004 seconds
[2016-12-18 00:06:26,539] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:26,539] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:11.539250
[2016-12-18 00:06:31,027] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:31,038] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:31,138] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,140] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,143] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:31,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:31,194] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,201] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,243] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,248] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:31,248] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,251] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,299] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,303] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,350] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,352] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,355] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:31,362] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,368] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,408] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,410] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,456] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,458] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,462] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:31,509] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:31,512] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:31,515] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:31,542] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:31,543] {jobs.py:744} INFO - Loop took: 0.520626 seconds
[2016-12-18 00:06:31,556] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:31,556] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:16.556679
[2016-12-18 00:06:36,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:36,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:36,126] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,130] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:36,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:36,189] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,191] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,237] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,238] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,239] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,240] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,241] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,243] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:36,292] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,294] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,297] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,339] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,341] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,344] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:36,345] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,347] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,349] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,398] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,400] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,403] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,426] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,428] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,431] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:36,477] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:36,479] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:36,482] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:36,506] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:36,506] {jobs.py:744} INFO - Loop took: 0.494566 seconds
[2016-12-18 00:06:36,517] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:36,517] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:21.517868
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:06:42,314] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:06:42,376] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:06:42,393] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:06:42,632] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:06:42,633] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:06:42,760] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:06:42,782] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:42,792] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:42,832] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,833] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,834] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,836] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,837] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:42,839] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:42,884] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,886] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,886] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,888] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,889] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:42,891] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:42,930] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,932] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,932] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,934] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,935] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:42,937] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:42,977] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:42,979] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:42,981] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:43,027] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:43,029] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:43,031] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:43,044] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:43,046] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:43,049] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:43,080] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:43,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:43,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:43,130] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:43,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:43,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:43,176] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:43,178] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:43,181] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:43,211] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:43,212] {jobs.py:744} INFO - Loop took: 0.435361 seconds
[2016-12-18 00:06:43,222] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:43,222] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:28.222570
[2016-12-18 00:06:47,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:47,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:47,128] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:47,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:47,183] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,188] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:47,190] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,193] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:47,232] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,237] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:47,238] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,240] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,242] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:47,287] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:47,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:47,292] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:47,409] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:47,409] {jobs.py:744} INFO - Loop took: 0.397539 seconds
[2016-12-18 00:06:47,419] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:47,419] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:32.419661
[2016-12-18 00:06:52,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:52,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:52,092] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:52,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:52,147] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,196] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,197] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,201] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:52,202] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,246] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,248] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,252] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,298] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:52,310] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,312] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,314] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,351] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,353] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,355] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,393] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,395] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,398] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:52,446] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:52,448] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:52,451] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:52,479] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:52,480] {jobs.py:744} INFO - Loop took: 0.468176 seconds
[2016-12-18 00:06:52,490] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:52,490] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:37.490458
[2016-12-18 00:06:57,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:06:57,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:06:57,133] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:57,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:57,193] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,194] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,199] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,247] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,249] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,249] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,251] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,252] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:57,254] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,301] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,303] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,306] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,353] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,355] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,358] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:57,370] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,372] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,375] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,409] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,411] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,414] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,460] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,465] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:06:57,514] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:06:57,517] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:06:57,520] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:06:57,556] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:06:57,556] {jobs.py:744} INFO - Loop took: 0.543854 seconds
[2016-12-18 00:06:57,566] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:06:57,567] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:42.567011
[2016-12-18 00:07:02,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:02,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:02,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,095] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:02,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:02,150] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,152] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,152] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,203] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:02,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,251] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,305] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,310] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:02,315] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,320] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,363] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,366] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,369] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,403] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,405] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,408] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:02,457] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:02,459] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:02,462] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:02,497] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:02,497] {jobs.py:744} INFO - Loop took: 0.485839 seconds
[2016-12-18 00:07:02,507] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:02,507] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:47.507665
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:07:08,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:07:08,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:07:08,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:07:08,640] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:07:08,640] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:07:08,767] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:07:08,790] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:08,799] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:08,839] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,839] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,841] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,841] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,844] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:08,844] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:08,895] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,897] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,897] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,899] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,900] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:08,902] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:08,947] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,949] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,949] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,951] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:08,952] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:08,954] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:08,996] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:08,998] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,001] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:09,052] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:09,054] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,057] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:09,059] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:09,061] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,064] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:09,103] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:09,105] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,107] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:09,137] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:09,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,142] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:09,184] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:09,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:09,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:09,211] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:09,212] {jobs.py:744} INFO - Loop took: 0.428041 seconds
[2016-12-18 00:07:09,229] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:09,229] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:54.229344
[2016-12-18 00:07:13,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:13,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:13,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:13,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:13,183] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,184] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,186] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,187] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:13,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:13,233] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,233] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,235] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,237] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:13,238] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:13,281] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:13,283] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:13,286] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:13,395] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:13,396] {jobs.py:744} INFO - Loop took: 0.384889 seconds
[2016-12-18 00:07:13,408] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:13,408] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:04:58.408753
[2016-12-18 00:07:18,024] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:18,033] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:18,075] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,075] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,076] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:18,080] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:18,130] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,132] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,178] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,180] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,185] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:18,224] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,226] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,229] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,271] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,273] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,275] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:18,287] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,292] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,321] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,326] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,374] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,376] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,379] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:18,423] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:18,425] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:18,427] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:18,458] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:18,459] {jobs.py:744} INFO - Loop took: 0.440224 seconds
[2016-12-18 00:07:18,490] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:18,491] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:03.491043
[2016-12-18 00:07:23,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:23,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:23,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,095] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:23,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:23,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,157] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,196] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:23,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,250] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,252] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,255] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,297] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,299] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:23,304] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,350] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,354] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,378] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,379] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,382] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:23,425] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:23,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:23,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:23,453] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:23,453] {jobs.py:744} INFO - Loop took: 0.441314 seconds
[2016-12-18 00:07:23,470] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:23,471] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:08.471083
[2016-12-18 00:07:28,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:28,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:28,081] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,081] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:28,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:28,140] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,145] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,191] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,194] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,197] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:28,243] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,248] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,294] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,296] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,299] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:28,327] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,329] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,332] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,349] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,353] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,418] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,420] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,423] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:28,472] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:28,474] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:28,477] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:28,515] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:28,515] {jobs.py:744} INFO - Loop took: 0.503542 seconds
[2016-12-18 00:07:28,525] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:28,525] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:13.525557
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:07:34,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:07:34,380] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:07:34,397] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:07:34,588] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:07:34,588] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:07:34,715] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:07:34,739] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:34,748] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:34,790] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,791] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,794] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,794] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:34,796] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,798] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:34,846] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,848] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,850] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,851] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:34,852] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,854] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:34,893] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,895] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,897] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,898] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:34,899] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,902] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:34,953] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:34,955] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:34,958] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:35,003] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:35,005] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:35,007] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:35,008] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:35,009] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:35,012] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:35,064] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:35,066] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:35,069] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:35,088] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:35,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:35,092] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:35,137] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:35,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:35,142] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:35,166] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:35,166] {jobs.py:744} INFO - Loop took: 0.43387 seconds
[2016-12-18 00:07:35,178] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:35,178] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:20.178695
[2016-12-18 00:07:39,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:39,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:39,079] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,082] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,084] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:39,087] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:39,134] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,136] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,136] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,138] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,139] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:39,141] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:39,186] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,188] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,189] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,191] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:39,194] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:39,236] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,238] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,241] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:39,289] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,294] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:39,300] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:39,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:39,305] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:39,406] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:39,407] {jobs.py:744} INFO - Loop took: 0.395371 seconds
[2016-12-18 00:07:39,419] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:39,419] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:24.419565
[2016-12-18 00:07:44,024] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:44,032] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:44,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,098] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:44,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:44,146] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:44,207] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,212] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,254] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,256] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,299] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,301] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,303] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,305] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:44,355] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,380] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,382] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,385] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:44,427] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:44,429] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:44,431] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:44,457] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:44,458] {jobs.py:744} INFO - Loop took: 0.439494 seconds
[2016-12-18 00:07:44,467] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:44,468] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:29.467993
[2016-12-18 00:07:49,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:49,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:49,073] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,075] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,077] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,078] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:49,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:49,126] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,130] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,172] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,174] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,176] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,177] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:49,178] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,221] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,223] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,226] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,268] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,269] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,272] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:49,278] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,283] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,320] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,324] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,359] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,361] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,363] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:49,406] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:49,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:49,410] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:49,434] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:49,434] {jobs.py:744} INFO - Loop took: 0.422917 seconds
[2016-12-18 00:07:49,444] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:49,444] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:34.444514
[2016-12-18 00:07:54,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:07:54,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:07:54,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:54,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:54,144] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,146] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,195] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:54,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,244] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,246] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,249] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,296] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,298] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,300] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:54,306] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,311] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,349] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,350] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,353] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,396] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,398] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,401] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:07:54,443] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:07:54,446] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:07:54,448] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:07:54,472] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:07:54,472] {jobs.py:744} INFO - Loop took: 0.460839 seconds
[2016-12-18 00:07:54,482] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:07:54,482] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:39.482523
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:08:00,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:08:00,380] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:08:00,397] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:08:00,604] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:08:00,604] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:08:00,731] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:08:00,755] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:00,764] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:00,805] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,805] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:00,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:00,861] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,862] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,862] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,864] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,865] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:00,867] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:00,909] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,911] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,914] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:00,924] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,925] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,928] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:00,972] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:00,974] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:00,977] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:01,031] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:01,032] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:01,032] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:01,034] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:01,035] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:01,036] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:01,089] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:01,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:01,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:01,115] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:01,117] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:01,120] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:01,170] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:01,172] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:01,175] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:01,201] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:01,201] {jobs.py:744} INFO - Loop took: 0.452868 seconds
[2016-12-18 00:08:01,240] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:01,241] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:46.241141
[2016-12-18 00:08:05,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:05,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:05,096] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,097] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:05,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:05,153] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,155] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,155] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,158] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:05,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:05,207] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,208] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,212] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:05,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:05,259] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:05,261] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:05,264] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:05,396] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:05,396] {jobs.py:744} INFO - Loop took: 0.381848 seconds
[2016-12-18 00:08:05,407] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:05,407] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:50.407405
[2016-12-18 00:08:10,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:10,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:10,092] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,096] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:10,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:10,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,150] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,152] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,198] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,204] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:10,211] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,216] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,264] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,266] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,269] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,326] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,326] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:10,328] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,332] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,396] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,398] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,401] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,425] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:10,479] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:10,481] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:10,484] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:10,511] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:10,512] {jobs.py:744} INFO - Loop took: 0.498714 seconds
[2016-12-18 00:08:10,526] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:10,526] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:05:55.526359
[2016-12-18 00:08:15,022] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:15,031] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:15,128] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:15,136] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,138] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,141] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:15,182] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,184] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,199] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,233] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,235] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,239] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:15,251] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,302] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,304] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,307] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,353] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,355] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,355] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,358] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:15,361] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,408] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,410] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,447] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,449] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,453] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:15,504] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:15,506] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:15,509] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:15,537] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:15,537] {jobs.py:744} INFO - Loop took: 0.521149 seconds
[2016-12-18 00:08:15,548] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:15,548] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:00.548587
[2016-12-18 00:08:20,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:20,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:20,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,093] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:20,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:20,147] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,202] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,207] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:20,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,256] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,308] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,310] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,314] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:20,332] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,334] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,337] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,367] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,369] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,372] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,425] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:20,481] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:20,483] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:20,486] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:20,517] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:20,517] {jobs.py:744} INFO - Loop took: 0.504634 seconds
[2016-12-18 00:08:20,527] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:20,527] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:05.527809
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:08:26,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:08:26,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:08:26,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:08:26,593] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:08:26,593] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:08:26,720] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:08:26,743] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:26,752] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:26,791] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,792] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,793] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,794] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,796] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:26,797] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:26,843] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,844] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,844] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,846] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,847] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:26,849] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:26,888] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,889] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,890] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,891] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,892] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:26,894] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:26,935] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,937] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,940] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:26,982] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,983] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,986] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:26,993] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:26,994] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:26,997] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:27,032] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:27,034] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:27,037] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:27,075] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:27,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:27,080] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:27,128] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:27,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:27,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:27,161] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:27,161] {jobs.py:744} INFO - Loop took: 0.424488 seconds
[2016-12-18 00:08:27,171] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:27,171] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:12.171270
[2016-12-18 00:08:31,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:31,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:31,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:31,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:31,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,144] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:31,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:31,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,196] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:31,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:31,251] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,255] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:31,307] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,309] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,312] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:31,317] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:31,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:31,323] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:31,436] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:31,436] {jobs.py:744} INFO - Loop took: 0.424792 seconds
[2016-12-18 00:08:31,445] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:31,446] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:16.446036
[2016-12-18 00:08:36,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:36,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:36,123] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,123] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,127] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:36,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:36,175] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,176] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,176] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,178] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,179] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,221] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,223] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,223] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,225] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,226] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:36,227] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,271] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,273] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,275] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,318] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,322] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:36,329] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,331] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,334] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,375] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,377] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,379] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,409] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,410] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:36,455] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:36,456] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:36,459] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:36,490] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:36,490] {jobs.py:744} INFO - Loop took: 0.479308 seconds
[2016-12-18 00:08:36,500] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:36,500] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:21.500534
[2016-12-18 00:08:41,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:41,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:41,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:41,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:41,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,144] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,189] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,193] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:41,193] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,239] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,241] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,244] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,285] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,287] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,290] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:41,297] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,302] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,337] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,339] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,341] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,376] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,378] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,380] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:41,423] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:41,424] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:41,427] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:41,451] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:41,451] {jobs.py:744} INFO - Loop took: 0.439657 seconds
[2016-12-18 00:08:41,461] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:41,461] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:26.461256
[2016-12-18 00:08:46,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:46,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:46,121] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,123] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,125] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,126] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:46,127] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:46,180] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,184] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,185] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,186] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,226] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,228] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,229] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,230] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:46,231] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,234] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,275] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,277] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,280] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,326] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:46,330] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,331] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,334] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,372] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,374] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,378] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,411] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,413] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,416] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:46,463] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:46,465] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:46,468] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:46,495] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:46,496] {jobs.py:744} INFO - Loop took: 0.484121 seconds
[2016-12-18 00:08:46,505] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:46,505] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:31.505911
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:08:52,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:08:52,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:08:52,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:08:52,640] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:08:52,640] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:08:52,767] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:08:52,790] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:52,800] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:52,840] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,841] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,842] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,842] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,845] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:52,845] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:52,893] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,894] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,895] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,897] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:52,897] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,900] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:52,938] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,940] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,942] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:52,948] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,950] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:52,953] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:52,996] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:52,997] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,000] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:53,041] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:53,043] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,044] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:53,046] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,046] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:53,049] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:53,092] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:53,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:53,122] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:53,124] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,127] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:53,171] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:53,173] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:53,175] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:53,199] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:53,199] {jobs.py:744} INFO - Loop took: 0.414945 seconds
[2016-12-18 00:08:53,209] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:53,209] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:38.209290
[2016-12-18 00:08:57,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:08:57,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:08:57,077] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,077] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:57,082] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:57,128] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,130] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:57,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:57,176] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,178] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,180] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:57,183] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:08:57,230] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:08:57,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:08:57,234] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:08:57,336] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:08:57,336] {jobs.py:744} INFO - Loop took: 0.325185 seconds
[2016-12-18 00:08:57,346] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:08:57,347] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:42.347128
[2016-12-18 00:09:02,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:02,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:02,087] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,087] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,092] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:02,092] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:02,152] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,153] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,155] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,203] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,207] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:02,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,259] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,261] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,264] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,307] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,309] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,312] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:02,316] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,318] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,321] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,363] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,365] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,368] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,395] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,397] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,400] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:02,444] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:02,446] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:02,449] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:02,474] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:02,474] {jobs.py:744} INFO - Loop took: 0.46183 seconds
[2016-12-18 00:09:02,483] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:02,484] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:47.484225
[2016-12-18 00:09:07,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:07,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:07,098] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,099] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:07,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:07,156] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,158] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,158] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,161] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,207] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,208] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,211] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,213] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:07,256] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,305] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,310] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:07,322] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,324] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,327] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,360] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,362] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,365] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,405] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,407] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,409] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:07,452] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:07,454] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:07,457] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:07,481] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:07,482] {jobs.py:744} INFO - Loop took: 0.470465 seconds
[2016-12-18 00:09:07,492] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:07,492] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:52.492234
[2016-12-18 00:09:12,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:12,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:12,074] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,075] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,076] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:12,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:12,126] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,128] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,174] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,176] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,176] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,178] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:12,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,223] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,225] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,227] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,269] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,271] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,274] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:12,281] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,283] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,286] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,322] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,324] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,327] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,358] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,360] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,363] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:12,405] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:12,407] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:12,410] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:12,437] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:12,437] {jobs.py:744} INFO - Loop took: 0.425481 seconds
[2016-12-18 00:09:12,447] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:12,447] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:06:57.447324
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:09:18,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:09:18,383] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:09:18,400] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:09:18,604] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:09:18,604] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:09:18,731] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:09:18,754] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:18,764] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:18,805] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,809] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:18,811] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,813] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:18,858] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,860] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,861] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:18,863] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,865] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:18,911] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,913] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,913] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,914] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,915] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:18,917] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:18,960] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:18,962] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:18,964] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:19,015] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:19,017] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:19,018] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:19,019] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:19,020] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:19,023] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:19,066] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:19,068] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:19,070] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:19,097] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:19,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:19,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:19,150] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:19,152] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:19,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:19,179] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:19,179] {jobs.py:744} INFO - Loop took: 0.431116 seconds
[2016-12-18 00:09:19,189] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:19,189] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:04.189486
[2016-12-18 00:09:23,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:23,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:23,087] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,092] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:23,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:23,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,141] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:23,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:23,195] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,195] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,200] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:23,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:23,244] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,248] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:23,299] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:23,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:23,304] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:23,390] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:23,390] {jobs.py:744} INFO - Loop took: 0.379648 seconds
[2016-12-18 00:09:23,400] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:23,400] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:08.400929
[2016-12-18 00:09:28,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:28,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:28,098] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,099] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:28,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:28,154] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,157] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,158] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,205] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,207] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,207] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,210] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:28,212] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,257] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,259] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,262] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,306] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,311] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:28,328] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,329] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,332] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,360] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,363] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,366] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,411] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,413] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,416] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:28,463] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:28,466] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:28,469] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:28,495] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:28,495] {jobs.py:744} INFO - Loop took: 0.480098 seconds
[2016-12-18 00:09:28,506] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:28,507] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:13.507034
[2016-12-18 00:09:33,015] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:33,023] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:33,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:33,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:33,141] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,146] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,191] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,193] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,196] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:33,197] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,245] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,249] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,294] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,296] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,299] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:33,302] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,304] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,307] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,352] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,354] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,357] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,384] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,385] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,388] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:33,431] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:33,433] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:33,436] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:33,466] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:33,466] {jobs.py:744} INFO - Loop took: 0.456777 seconds
[2016-12-18 00:09:33,476] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:33,476] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:18.476552
[2016-12-18 00:09:38,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:38,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:38,073] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,075] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,078] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:38,078] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,083] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:38,128] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,135] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,137] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,140] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,175] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,180] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:38,183] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,231] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,233] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,236] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,278] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,283] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:38,283] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,285] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,288] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,330] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,332] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,335] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,363] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,365] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:38,411] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:38,413] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:38,416] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:38,441] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:38,441] {jobs.py:744} INFO - Loop took: 0.429837 seconds
[2016-12-18 00:09:38,452] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:38,452] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:23.452314
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:09:44,318] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:09:44,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:09:44,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:09:44,592] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:09:44,592] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:09:44,721] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:09:44,744] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:44,754] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:44,801] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,802] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,803] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,804] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,806] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:44,807] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:44,862] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,863] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,864] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,865] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,867] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:44,868] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:44,911] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,913] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,913] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,915] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,916] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:44,918] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:44,965] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:44,967] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:44,969] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:45,014] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:45,016] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:45,019] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:45,029] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:45,031] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:45,034] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:45,071] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:45,072] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:45,076] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:45,120] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:45,122] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:45,125] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:45,175] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:45,176] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:45,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:45,207] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:45,207] {jobs.py:744} INFO - Loop took: 0.469667 seconds
[2016-12-18 00:09:45,220] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:45,220] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:30.220851
[2016-12-18 00:09:49,015] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:49,024] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:49,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,090] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:49,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:49,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,145] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:49,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:49,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,192] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,196] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:49,197] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:49,238] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:49,240] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:49,243] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:49,350] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:49,350] {jobs.py:744} INFO - Loop took: 0.340641 seconds
[2016-12-18 00:09:49,360] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:49,360] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:34.360951
[2016-12-18 00:09:54,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:54,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:54,125] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,127] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,129] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:54,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:54,178] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,180] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,183] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,183] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,225] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,227] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,230] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:54,240] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,242] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,244] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,286] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,288] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,291] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,334] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,335] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,336] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,337] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,338] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,340] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:54,387] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,388] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,391] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,412] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,414] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,417] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:54,459] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:54,461] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:54,464] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:54,487] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:54,487] {jobs.py:744} INFO - Loop took: 0.476117 seconds
[2016-12-18 00:09:54,497] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:54,497] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:39.497888
[2016-12-18 00:09:59,025] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:09:59,034] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:09:59,156] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,157] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,158] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,161] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:59,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:59,211] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,215] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,217] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,219] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,222] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,257] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,259] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,262] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,274] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,276] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,280] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:59,305] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,309] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,355] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,360] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:59,401] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,403] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,407] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,407] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,495] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,497] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,500] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:09:59,549] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:09:59,551] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:09:59,554] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:09:59,582] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:09:59,582] {jobs.py:744} INFO - Loop took: 0.562799 seconds
[2016-12-18 00:09:59,592] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:09:59,592] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:44.592755
[2016-12-18 00:10:04,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:04,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:04,081] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,082] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,084] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:04,087] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:04,138] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,140] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,141] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,198] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,201] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,203] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:04,206] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,255] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,258] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,304] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,306] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,309] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:04,314] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,319] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,360] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,362] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,365] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,398] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,400] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,403] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:04,447] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:04,449] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:04,451] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:04,477] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:04,477] {jobs.py:744} INFO - Loop took: 0.465495 seconds
[2016-12-18 00:10:04,487] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:04,488] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:49.488132
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:10:10,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:10:10,383] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:10:10,400] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:10:10,604] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:10:10,604] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:10:10,734] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:10:10,758] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:10,767] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:10,807] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,808] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,809] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,810] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,812] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:10,813] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:10,860] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,861] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,863] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,864] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:10,866] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:10,906] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,908] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,908] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,910] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,911] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:10,913] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:10,955] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:10,957] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:10,960] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:11,009] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:11,011] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:11,014] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:11,014] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:11,015] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:11,018] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:11,065] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:11,067] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:11,071] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:11,097] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:11,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:11,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:11,152] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:11,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:11,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:11,185] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:11,185] {jobs.py:744} INFO - Loop took: 0.433466 seconds
[2016-12-18 00:10:11,195] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:11,195] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:07:56.195392
[2016-12-18 00:10:15,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:15,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:15,078] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,079] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,083] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:15,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:15,132] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,134] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,134] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,136] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:15,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:15,183] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,183] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,187] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:15,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:15,230] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,235] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:15,278] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:15,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:15,282] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:15,347] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:15,347] {jobs.py:744} INFO - Loop took: 0.336034 seconds
[2016-12-18 00:10:15,357] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:15,357] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:00.357260
[2016-12-18 00:10:20,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:20,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:20,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:20,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:20,140] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,145] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,188] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,189] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,190] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,192] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:20,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,247] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,248] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,252] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,298] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,300] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,303] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,306] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:20,357] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,359] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,362] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,385] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,387] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,390] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:20,438] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:20,440] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:20,443] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:20,471] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:20,471] {jobs.py:744} INFO - Loop took: 0.458218 seconds
[2016-12-18 00:10:20,481] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:20,482] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:05.482090
[2016-12-18 00:10:25,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:25,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:25,089] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:25,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:25,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,147] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,191] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,193] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,195] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:25,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,301] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,303] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,306] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:25,311] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,313] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,317] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,360] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,362] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,364] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,400] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,402] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,405] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:25,462] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:25,464] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:25,468] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:25,496] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:25,496] {jobs.py:744} INFO - Loop took: 0.484136 seconds
[2016-12-18 00:10:25,506] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:25,506] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:10.506763
[2016-12-18 00:10:30,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:30,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:30,129] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,129] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:30,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:30,183] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,185] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,187] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,188] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,190] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,232] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,235] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,237] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,237] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:30,240] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,287] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,291] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,340] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,345] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:30,354] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,356] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,359] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,397] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,399] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,403] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,443] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,445] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,448] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:30,497] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:30,499] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:30,503] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:30,532] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:30,532] {jobs.py:744} INFO - Loop took: 0.518418 seconds
[2016-12-18 00:10:30,542] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:30,542] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:15.542427
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:10:36,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:10:36,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:10:36,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:10:36,590] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:10:36,590] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:10:36,716] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:10:36,739] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:36,749] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:36,802] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,802] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,804] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,804] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,806] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:36,807] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:36,857] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,859] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,859] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:36,864] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:36,908] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,910] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,910] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,911] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,912] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:36,914] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:36,960] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:36,963] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:36,965] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:37,014] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:37,016] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:37,019] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:37,023] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:37,025] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:37,027] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:37,071] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:37,073] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:37,076] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:37,118] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:37,121] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:37,124] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:37,172] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:37,174] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:37,178] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:37,207] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:37,207] {jobs.py:744} INFO - Loop took: 0.474209 seconds
[2016-12-18 00:10:37,222] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:37,222] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:22.222749
[2016-12-18 00:10:41,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:41,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:41,122] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,124] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,127] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:41,137] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,142] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:41,185] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,186] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:41,191] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,193] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:41,234] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,239] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:41,239] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,241] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,243] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:41,291] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:41,293] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:41,295] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:41,397] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:41,397] {jobs.py:744} INFO - Loop took: 0.386046 seconds
[2016-12-18 00:10:41,406] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:41,407] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:26.407132
[2016-12-18 00:10:46,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:46,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:46,123] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:46,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:46,175] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,183] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,225] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,227] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,230] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:46,241] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,243] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,246] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,293] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,295] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,298] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,348] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,349] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,349] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,352] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,354] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:46,405] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,407] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,409] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,428] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,430] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,433] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:46,476] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:46,478] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:46,480] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:46,504] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:46,504] {jobs.py:744} INFO - Loop took: 0.492488 seconds
[2016-12-18 00:10:46,522] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:46,522] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:31.522485
[2016-12-18 00:10:51,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:51,030] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:51,101] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,101] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,106] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:51,106] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:51,161] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,163] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,165] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,167] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,210] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,212] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,214] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,215] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:51,216] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,219] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,267] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,269] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,272] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,319] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,324] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:51,342] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,343] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,347] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,376] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,378] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,381] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,431] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,433] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,436] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:51,490] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:51,492] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:51,495] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:51,523] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:51,523] {jobs.py:744} INFO - Loop took: 0.507461 seconds
[2016-12-18 00:10:51,533] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:51,533] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:36.533901
[2016-12-18 00:10:56,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:10:56,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:10:56,090] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,095] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:56,097] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,100] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:56,156] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,161] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,204] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,208] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,209] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:56,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,265] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,267] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,270] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,315] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,317] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,317] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,320] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:56,322] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,374] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,376] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,379] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,406] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:10:56,460] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:10:56,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:10:56,465] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:10:56,494] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:10:56,494] {jobs.py:744} INFO - Loop took: 0.482739 seconds
[2016-12-18 00:10:56,505] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:10:56,505] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:41.505259
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:11:02,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:11:02,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:11:02,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:11:02,591] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:11:02,592] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:11:02,724] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:11:02,749] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:02,759] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:02,800] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,801] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,803] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,803] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,806] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:02,806] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:02,857] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,859] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,859] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:02,863] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:02,904] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,906] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,906] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,908] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,909] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:02,911] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:02,953] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:02,955] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:02,958] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:03,006] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:03,008] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:03,011] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:03,019] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:03,021] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:03,025] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:03,063] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:03,065] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:03,068] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:03,112] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:03,114] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:03,116] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:03,161] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:03,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:03,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:03,190] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:03,190] {jobs.py:744} INFO - Loop took: 0.447799 seconds
[2016-12-18 00:11:03,200] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:03,201] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:48.201182
[2016-12-18 00:11:07,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:07,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:07,091] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:07,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:07,144] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,147] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:07,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:07,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,207] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:07,208] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,211] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:07,254] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,256] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:07,309] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:07,311] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:07,314] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:07,367] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:07,367] {jobs.py:744} INFO - Loop took: 0.356648 seconds
[2016-12-18 00:11:07,384] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:07,384] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:52.384940
[2016-12-18 00:11:12,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:12,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:12,079] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,081] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:12,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:12,137] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,189] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:12,238] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,240] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,243] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,289] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,294] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:12,316] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,318] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,321] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,347] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,349] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,352] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,402] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,404] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,407] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:12,459] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:12,461] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:12,464] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:12,492] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:12,492] {jobs.py:744} INFO - Loop took: 0.480617 seconds
[2016-12-18 00:11:12,502] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:12,502] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:08:57.502810
[2016-12-18 00:11:17,028] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:17,037] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:17,136] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,138] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,141] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:17,141] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:17,193] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,198] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,202] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,207] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,245] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,250] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:17,256] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,308] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,310] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,313] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,362] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,364] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,366] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:17,370] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,421] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,423] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,426] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,453] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,455] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,458] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:17,506] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:17,508] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:17,511] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:17,542] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:17,542] {jobs.py:744} INFO - Loop took: 0.519128 seconds
[2016-12-18 00:11:17,552] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:17,553] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:02.553122
[2016-12-18 00:11:22,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:22,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:22,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:22,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:22,155] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,157] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,205] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,208] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,208] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,210] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:22,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,260] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,262] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,266] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,313] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,315] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,319] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:22,324] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,326] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,329] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,368] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,370] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,373] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,414] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,416] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,419] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:22,473] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:22,475] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:22,478] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:22,508] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:22,508] {jobs.py:744} INFO - Loop took: 0.497257 seconds
[2016-12-18 00:11:22,518] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:22,518] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:07.518798
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:11:28,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:11:28,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:11:28,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:11:28,604] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:11:28,604] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:11:28,731] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:11:28,754] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:28,763] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:28,807] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,809] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,811] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,812] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:28,813] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,816] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:28,861] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,862] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,865] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,865] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:28,867] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,869] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:28,908] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,910] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,912] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,913] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:28,914] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,917] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:28,962] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:28,964] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:28,967] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:29,014] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:29,016] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:29,018] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:29,033] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:29,035] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:29,039] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:29,068] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:29,070] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:29,073] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:29,124] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:29,126] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:29,129] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:29,178] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:29,180] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:29,183] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:29,211] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:29,211] {jobs.py:744} INFO - Loop took: 0.462832 seconds
[2016-12-18 00:11:29,223] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:29,223] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:14.223257
[2016-12-18 00:11:33,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:33,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:33,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,096] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:33,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:33,151] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,153] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,155] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:33,158] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:33,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,203] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,206] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:33,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:33,256] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:33,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:33,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:33,380] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:33,380] {jobs.py:744} INFO - Loop took: 0.368566 seconds
[2016-12-18 00:11:33,390] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:33,390] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:18.390747
[2016-12-18 00:11:38,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:38,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:38,135] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,135] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,137] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,137] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,140] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:38,140] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:38,194] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,196] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,248] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,249] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,250] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,252] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,252] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:38,255] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,298] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,302] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,345] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,346] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,349] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:38,371] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,373] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,376] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,402] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,404] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,406] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,452] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,454] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,456] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:38,507] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:38,509] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:38,511] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:38,536] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:38,536] {jobs.py:744} INFO - Loop took: 0.524655 seconds
[2016-12-18 00:11:38,546] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:38,546] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:23.546268
[2016-12-18 00:11:43,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:43,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:43,132] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,132] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,134] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,134] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:43,137] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:43,185] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,187] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,187] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,188] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,191] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,231] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,232] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,235] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:43,237] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,279] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,283] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,326] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,328] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,331] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:43,337] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,339] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,342] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,383] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,385] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,387] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,415] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,417] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,420] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:43,461] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:43,463] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:43,466] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:43,496] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:43,496] {jobs.py:744} INFO - Loop took: 0.484718 seconds
[2016-12-18 00:11:43,506] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:43,506] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:28.506352
[2016-12-18 00:11:48,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:48,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:48,122] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,124] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,126] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,127] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:48,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:48,175] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,187] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,189] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,192] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,222] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,223] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,226] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:48,234] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,239] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,281] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,282] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,285] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,327] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,329] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,329] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,331] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,331] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:48,334] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,380] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,382] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,385] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,409] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,411] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:48,455] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:48,457] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:48,460] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:48,483] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:48,483] {jobs.py:744} INFO - Loop took: 0.471084 seconds
[2016-12-18 00:11:48,499] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:48,500] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:33.500184
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:11:54,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:11:54,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:11:54,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:11:54,638] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:11:54,638] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:11:54,767] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:11:54,790] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:54,800] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:54,840] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,840] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,842] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,842] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,844] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:54,845] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:54,898] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,900] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,900] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,901] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,902] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:54,904] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:54,947] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,949] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,949] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:54,951] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:54,952] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:54,954] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:55,006] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,008] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,011] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:55,059] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,061] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,064] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:55,072] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,073] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,077] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:55,117] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,119] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,122] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:55,160] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,162] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:55,220] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:55,222] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:55,225] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:55,253] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:55,253] {jobs.py:744} INFO - Loop took: 0.46954 seconds
[2016-12-18 00:11:55,263] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:55,263] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:40.263933
[2016-12-18 00:11:59,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:11:59,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:11:59,087] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,089] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,091] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,092] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:59,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:59,140] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:59,152] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:59,187] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,189] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,192] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:11:59,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:59,247] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:11:59,249] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:11:59,252] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:11:59,347] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:11:59,347] {jobs.py:744} INFO - Loop took: 0.336069 seconds
[2016-12-18 00:11:59,359] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:11:59,359] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:44.359698
[2016-12-18 00:12:04,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:04,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:04,090] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:04,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:04,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,201] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,203] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,206] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:04,208] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,254] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,257] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,306] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,311] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:04,314] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,319] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,359] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,361] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,364] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,396] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,398] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,400] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:04,443] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:04,445] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:04,448] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:04,471] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:04,472] {jobs.py:744} INFO - Loop took: 0.460198 seconds
[2016-12-18 00:12:04,484] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:04,484] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:49.484498
[2016-12-18 00:12:09,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:09,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:09,066] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,068] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,071] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:09,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:09,120] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,121] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,124] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,165] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,167] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,169] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,194] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,199] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:09,211] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,216] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,257] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,259] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,262] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:09,302] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,303] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,306] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,308] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,310] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,312] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,379] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,381] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,384] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:09,425] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:09,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:09,429] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:09,454] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:09,454] {jobs.py:744} INFO - Loop took: 0.442737 seconds
[2016-12-18 00:12:09,464] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:09,465] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:54.465033
[2016-12-18 00:12:14,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:14,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:14,075] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,075] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,079] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:14,080] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:14,129] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,131] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,175] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,177] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,180] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:14,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,224] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,226] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,229] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,272] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,274] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,277] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:14,282] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,284] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,287] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,329] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,331] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,333] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,362] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,366] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:14,408] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:14,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:14,412] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:14,442] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:14,442] {jobs.py:744} INFO - Loop took: 0.431074 seconds
[2016-12-18 00:12:14,452] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:14,452] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:09:59.452698
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:12:20,327] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:12:20,389] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:12:20,406] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:12:20,600] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:12:20,600] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:12:20,729] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:12:20,753] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:20,763] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:20,805] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,805] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,809] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:20,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:20,857] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,859] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,860] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,862] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:20,865] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:20,906] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,907] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,908] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,910] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,910] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:20,913] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:20,961] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:20,963] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:20,966] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:21,011] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:21,013] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:21,013] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:21,015] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:21,016] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:21,018] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:21,070] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:21,072] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:21,075] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:21,105] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:21,107] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:21,110] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:21,160] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:21,162] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:21,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:21,194] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:21,194] {jobs.py:744} INFO - Loop took: 0.447014 seconds
[2016-12-18 00:12:21,205] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:21,205] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:06.205252
[2016-12-18 00:12:25,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:25,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:25,092] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,093] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:25,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:25,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,147] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:25,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:25,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,194] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:25,199] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:25,241] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:25,243] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:25,245] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:25,353] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:25,353] {jobs.py:744} INFO - Loop took: 0.341946 seconds
[2016-12-18 00:12:25,363] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:25,363] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:10.363389
[2016-12-18 00:12:30,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:30,030] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:30,080] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,083] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:30,085] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,088] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:30,136] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,137] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,138] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,140] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,190] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,192] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,195] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:30,196] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,245] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,250] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,298] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,300] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,304] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:30,319] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,324] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,358] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,360] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,363] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,413] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,415] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,418] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:30,476] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:30,478] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:30,481] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:30,510] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:30,510] {jobs.py:744} INFO - Loop took: 0.495967 seconds
[2016-12-18 00:12:30,520] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:30,521] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:15.521074
[2016-12-18 00:12:35,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:35,030] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:35,080] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,081] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:35,087] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:35,133] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,141] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,180] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,184] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:35,229] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,230] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,233] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,283] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,285] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,288] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:35,314] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,319] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,335] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,337] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,339] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,393] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,395] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,398] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:35,441] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:35,443] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:35,445] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:35,469] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:35,470] {jobs.py:744} INFO - Loop took: 0.453892 seconds
[2016-12-18 00:12:35,479] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:35,479] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:20.479820
[2016-12-18 00:12:40,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:40,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:40,137] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,139] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,142] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:40,142] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:40,190] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,192] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,196] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,240] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,242] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,245] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:40,245] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,249] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,291] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,292] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,295] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,344] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,346] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,347] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,349] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:40,349] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,352] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,397] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,400] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,403] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,426] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,428] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,431] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:40,473] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:40,474] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:40,477] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:40,500] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:40,501] {jobs.py:744} INFO - Loop took: 0.48706 seconds
[2016-12-18 00:12:40,510] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:40,511] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:25.511113
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:12:46,326] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:12:46,388] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:12:46,406] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:12:46,620] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:12:46,620] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:12:46,749] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:12:46,773] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:46,783] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:46,824] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,824] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,826] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,826] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,829] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:46,829] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:46,877] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,879] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,881] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,881] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:46,883] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,886] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:46,924] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,925] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,928] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,928] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:46,930] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,932] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:46,976] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:46,978] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:46,981] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:47,022] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:47,024] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:47,027] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:47,031] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:47,034] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:47,082] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:47,084] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:47,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:47,241] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:47,318] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:47,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:47,322] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:47,373] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:47,375] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:47,378] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:47,402] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:47,403] {jobs.py:744} INFO - Loop took: 0.636078 seconds
[2016-12-18 00:12:47,412] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:47,413] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:32.413097
[2016-12-18 00:12:51,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:51,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:51,096] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,100] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:51,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,106] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:51,151] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,156] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:51,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:51,200] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:51,207] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,209] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,212] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:51,258] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:51,260] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:51,262] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:51,367] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:51,367] {jobs.py:744} INFO - Loop took: 0.355872 seconds
[2016-12-18 00:12:51,378] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:51,378] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:36.378786
[2016-12-18 00:12:56,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:12:56,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:12:56,095] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,100] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:56,102] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,105] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:56,151] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,155] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,202] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,206] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,207] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:56,208] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,212] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,258] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,260] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,263] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,313] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,315] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,319] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:56,324] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,326] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,329] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,367] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,369] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,373] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,419] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,421] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,425] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:12:56,475] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:12:56,477] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:12:56,480] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:12:56,507] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:12:56,508] {jobs.py:744} INFO - Loop took: 0.49607 seconds
[2016-12-18 00:12:56,518] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:12:56,518] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:41.518894
[2016-12-18 00:13:01,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:01,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:01,115] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,117] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,117] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,119] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,120] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:01,122] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:01,173] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,175] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,177] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,178] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,179] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,182] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,226] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,228] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,230] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,232] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,234] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,235] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:01,284] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,286] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,289] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,335] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,337] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,340] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:01,358] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,360] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,364] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,393] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,395] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,397] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,460] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,466] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:01,522] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:01,524] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:01,527] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:01,558] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:01,558] {jobs.py:744} INFO - Loop took: 0.546956 seconds
[2016-12-18 00:13:01,569] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:01,569] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:46.569700
[2016-12-18 00:13:06,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:06,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:06,128] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,128] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,130] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:06,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:06,183] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,185] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,185] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,187] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,187] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,189] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,229] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,231] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,231] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,233] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,234] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:06,235] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,277] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,279] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,281] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,322] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,324] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,327] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:06,336] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,338] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,341] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,373] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,375] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,414] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,416] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,419] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:06,467] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:06,469] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:06,472] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:06,500] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:06,500] {jobs.py:744} INFO - Loop took: 0.488897 seconds
[2016-12-18 00:13:06,514] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:06,514] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:51.514530
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:13:12,322] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:13:12,384] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:13:12,401] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:13:12,595] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:13:12,595] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:13:12,723] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:13:12,746] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:12,756] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:12,796] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,797] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,798] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,799] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,801] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:12,802] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:12,850] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,852] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,854] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:12,858] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,860] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:12,901] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,903] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,904] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,906] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:12,906] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,909] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:12,949] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,951] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:12,953] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:12,996] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:12,998] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:13,000] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:13,015] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:13,017] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:13,020] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:13,053] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:13,056] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:13,059] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:13,104] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:13,106] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:13,109] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:13,159] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:13,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:13,164] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:13,192] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:13,192] {jobs.py:744} INFO - Loop took: 0.452422 seconds
[2016-12-18 00:13:13,202] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:13,203] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:10:58.203040
[2016-12-18 00:13:17,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:17,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:17,093] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,095] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,098] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,098] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:17,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:17,146] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:17,152] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:17,199] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,203] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:17,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:17,251] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:17,300] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:17,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:17,305] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:17,359] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:17,359] {jobs.py:744} INFO - Loop took: 0.348496 seconds
[2016-12-18 00:13:17,369] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:17,369] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:02.369661
[2016-12-18 00:13:22,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:22,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:22,073] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,075] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,078] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,078] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:22,080] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,082] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:22,126] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,128] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,139] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,144] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,174] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,175] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,178] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:22,186] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,188] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,190] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,232] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,236] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,280] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,282] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,283] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,285] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:22,285] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,289] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,331] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,333] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,335] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,363] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,365] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:22,410] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:22,412] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:22,414] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:22,438] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:22,438] {jobs.py:744} INFO - Loop took: 0.426862 seconds
[2016-12-18 00:13:22,457] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:22,457] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:07.457842
[2016-12-18 00:13:27,016] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:27,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:27,085] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,087] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,089] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,090] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:27,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:27,140] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,148] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,200] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,202] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:27,206] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,254] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,257] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,313] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,315] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,318] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:27,322] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,324] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,327] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,371] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,373] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,416] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,418] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,421] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:27,469] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:27,471] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:27,474] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:27,502] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:27,502] {jobs.py:744} INFO - Loop took: 0.490881 seconds
[2016-12-18 00:13:27,512] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:27,512] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:12.512489
[2016-12-18 00:13:32,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:32,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:32,084] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,084] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,086] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,086] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,089] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:32,089] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:32,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,145] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,145] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,195] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:32,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,247] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,249] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,252] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,300] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,302] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,305] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:32,314] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,319] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,357] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,359] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,362] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,407] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,412] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:32,460] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:32,462] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:32,466] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:32,493] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:32,494] {jobs.py:744} INFO - Loop took: 0.482224 seconds
[2016-12-18 00:13:32,506] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:32,506] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:17.506888
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:13:38,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:13:38,383] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:13:38,400] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:13:38,609] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:13:38,610] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:13:38,739] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:13:38,762] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:38,772] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:38,821] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,821] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,823] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,824] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,825] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:38,826] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:38,874] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,876] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,876] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,878] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,879] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:38,881] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:38,928] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,929] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,930] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,931] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,932] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:38,934] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:38,978] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:38,980] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:38,983] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:39,034] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:39,035] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:39,038] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:39,038] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:39,039] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:39,042] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:39,085] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:39,087] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:39,090] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:39,117] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:39,119] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:39,122] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:39,172] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:39,174] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:39,176] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:39,200] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:39,200] {jobs.py:744} INFO - Loop took: 0.444884 seconds
[2016-12-18 00:13:39,211] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:39,211] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:24.211399
[2016-12-18 00:13:43,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:43,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:43,097] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,098] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:43,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:43,152] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,153] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,155] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:43,158] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:43,202] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,203] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,205] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,207] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:43,207] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:43,251] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:43,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:43,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:43,363] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:43,364] {jobs.py:744} INFO - Loop took: 0.351257 seconds
[2016-12-18 00:13:43,373] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:43,374] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:28.374175
[2016-12-18 00:13:48,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:48,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:48,075] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,077] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,080] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:48,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,085] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:48,130] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,142] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,180] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,181] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,184] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:48,192] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,197] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,240] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,242] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,244] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,288] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,289] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,292] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:48,294] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,296] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,299] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,337] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,339] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,342] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,372] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,374] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:48,424] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:48,426] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:48,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:48,458] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:48,458] {jobs.py:744} INFO - Loop took: 0.446394 seconds
[2016-12-18 00:13:48,468] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:48,468] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:33.468320
[2016-12-18 00:13:53,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:53,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:53,086] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,088] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,090] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,091] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:53,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:53,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,149] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,152] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,194] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,196] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,199] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:53,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,242] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,244] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,246] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,289] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,291] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,293] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:53,304] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,305] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,309] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,340] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,344] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,391] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,393] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,396] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:53,438] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:53,440] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:53,443] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:53,474] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:53,474] {jobs.py:744} INFO - Loop took: 0.463203 seconds
[2016-12-18 00:13:53,485] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:53,486] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:38.486213
[2016-12-18 00:13:58,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:13:58,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:13:58,129] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:58,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:58,181] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,183] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,187] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,189] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,192] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,227] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,229] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,232] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:58,233] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,235] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,238] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,280] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,282] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,284] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,330] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,331] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,334] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,334] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:58,336] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,339] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,389] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,391] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,393] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,417] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,419] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,421] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:13:58,464] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:13:58,466] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:13:58,469] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:13:58,500] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:13:58,500] {jobs.py:744} INFO - Loop took: 0.487108 seconds
[2016-12-18 00:13:58,510] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:13:58,510] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:43.510915
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:14:04,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:14:04,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:14:04,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:14:04,608] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:14:04,608] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:14:04,735] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:14:04,758] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:04,767] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:04,807] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,807] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,808] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,809] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,811] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:04,812] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:04,865] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,867] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,869] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,870] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:04,871] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,873] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:04,920] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,922] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,924] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,924] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:04,926] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,929] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:04,972] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:04,974] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:04,976] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:05,024] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:05,026] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:05,029] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:05,052] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:05,054] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:05,057] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:05,083] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:05,085] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:05,087] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:05,157] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:05,159] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:05,162] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:05,211] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:05,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:05,216] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:05,244] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:05,244] {jobs.py:744} INFO - Loop took: 0.492566 seconds
[2016-12-18 00:14:05,254] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:05,254] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:50.254612
[2016-12-18 00:14:09,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:09,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:09,091] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,093] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:09,096] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:09,144] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,145] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:09,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:09,194] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,196] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:09,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:09,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:09,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:09,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:09,375] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:09,375] {jobs.py:744} INFO - Loop took: 0.363951 seconds
[2016-12-18 00:14:09,393] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:09,393] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:54.393918
[2016-12-18 00:14:14,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:14,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:14,090] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:14,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:14,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,148] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,192] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,197] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:14,197] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,202] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,299] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,304] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:14,318] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,323] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,354] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,356] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,358] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,408] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,410] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:14,461] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:14,463] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:14,467] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:14,494] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:14,494] {jobs.py:744} INFO - Loop took: 0.482874 seconds
[2016-12-18 00:14:14,504] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:14,505] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:11:59.505110
[2016-12-18 00:14:19,023] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:19,031] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:19,097] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,102] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:19,104] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,107] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:19,163] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,165] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,165] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,167] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,168] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,170] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,214] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,215] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,216] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,218] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,218] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:19,221] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,269] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,271] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,274] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,327] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:19,344] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,346] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,350] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,380] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,382] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,385] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,436] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,439] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,442] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:19,491] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:19,493] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:19,497] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:19,525] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:19,525] {jobs.py:744} INFO - Loop took: 0.508409 seconds
[2016-12-18 00:14:19,535] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:19,536] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:04.536050
[2016-12-18 00:14:24,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:24,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:24,125] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,127] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,129] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,130] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:24,131] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,134] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:24,188] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,189] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,190] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,193] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,240] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,242] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,242] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,244] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,244] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,247] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:24,291] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,293] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,296] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,343] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,345] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,348] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:24,367] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,369] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,372] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,398] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,399] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,402] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,459] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,461] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,464] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:24,510] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:24,512] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:24,515] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:24,538] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:24,538] {jobs.py:744} INFO - Loop took: 0.526682 seconds
[2016-12-18 00:14:24,548] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:24,548] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:09.548496
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:14:30,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:14:30,380] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:14:30,398] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:14:30,601] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:14:30,602] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:14:30,732] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:14:30,755] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:30,764] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:30,806] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,806] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,808] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,808] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:30,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:30,857] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,859] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,859] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,862] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:30,863] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:30,904] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,905] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,906] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,907] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,909] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:30,910] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:30,951] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,953] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:30,956] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:30,997] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:30,999] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:31,001] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:31,012] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:31,013] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:31,016] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:31,048] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:31,050] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:31,053] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:31,101] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:31,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:31,106] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:31,154] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:31,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:31,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:31,186] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:31,187] {jobs.py:744} INFO - Loop took: 0.4379 seconds
[2016-12-18 00:14:31,202] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:31,202] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:16.202544
[2016-12-18 00:14:35,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:35,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:35,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:35,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:35,189] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:35,200] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:35,237] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,239] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,242] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:35,250] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,252] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,254] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:35,302] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,304] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,307] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:35,349] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:35,351] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:35,354] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:35,410] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:35,410] {jobs.py:744} INFO - Loop took: 0.395744 seconds
[2016-12-18 00:14:35,420] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:35,420] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:20.420325
[2016-12-18 00:14:40,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:40,032] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:40,097] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,102] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:40,104] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,107] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:40,155] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,157] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,160] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,161] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,167] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,217] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,218] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,218] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,220] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,221] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:40,223] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,265] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,267] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,270] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,319] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,321] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,324] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:40,324] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,326] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,329] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,371] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,373] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,375] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,402] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,404] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,407] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:40,449] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:40,450] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:40,453] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:40,476] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:40,477] {jobs.py:744} INFO - Loop took: 0.462291 seconds
[2016-12-18 00:14:40,488] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:40,488] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:25.488385
[2016-12-18 00:14:45,023] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:45,031] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:45,098] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,099] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:45,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:45,159] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,161] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,166] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,210] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,212] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,212] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,214] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,215] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:45,217] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,262] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,264] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,266] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,314] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,319] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:45,340] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,342] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,345] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,370] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,372] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,375] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,432] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,434] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,437] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:45,486] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:45,488] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:45,491] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:45,526] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:45,526] {jobs.py:744} INFO - Loop took: 0.509073 seconds
[2016-12-18 00:14:45,535] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:45,536] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:30.536221
[2016-12-18 00:14:50,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:50,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:50,123] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,127] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:50,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,131] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:50,175] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,177] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,180] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,180] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,181] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,184] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,220] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,222] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,225] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:50,234] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,238] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,283] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,285] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,288] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,334] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,336] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,339] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:50,346] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,348] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,351] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,390] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,392] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,395] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,438] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,439] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,443] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:50,491] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:50,492] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:50,496] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:50,523] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:50,523] {jobs.py:744} INFO - Loop took: 0.510312 seconds
[2016-12-18 00:14:50,533] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:50,534] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:35.534165
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:14:56,323] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:14:56,385] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:14:56,402] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:14:56,602] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:14:56,603] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:14:56,730] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:14:56,753] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:14:56,762] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:14:56,802] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,803] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,804] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,805] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,806] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:56,807] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:56,854] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,855] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,856] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,858] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:56,858] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,861] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:56,899] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,901] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,901] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,903] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,903] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:56,905] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:56,955] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:56,957] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:56,959] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:57,001] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:57,002] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:57,005] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:57,006] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:57,008] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:57,011] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:57,059] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:57,061] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:57,064] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:57,088] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:57,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:57,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:14:57,141] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:14:57,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:14:57,146] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:14:57,173] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:14:57,173] {jobs.py:744} INFO - Loop took: 0.426757 seconds
[2016-12-18 00:14:57,183] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:14:57,184] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:42.184026
[2016-12-18 00:15:01,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:01,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:01,090] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,092] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,095] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:01,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:01,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,151] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:01,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:01,204] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,204] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,209] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:01,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:01,258] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,261] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,263] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:01,314] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,316] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,319] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:01,337] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:01,340] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:01,343] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:01,467] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:01,467] {jobs.py:744} INFO - Loop took: 0.45597 seconds
[2016-12-18 00:15:01,478] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:01,478] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:46.478330
[2016-12-18 00:15:06,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:06,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:06,076] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,078] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,080] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:06,082] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,086] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:06,139] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,141] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,141] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,143] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,143] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,187] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,189] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,189] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,192] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:06,194] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,245] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,250] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,295] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,296] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,297] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,299] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,299] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:06,302] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,356] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,358] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,361] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,380] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,382] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,385] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:06,433] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:06,435] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:06,438] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:06,466] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:06,466] {jobs.py:744} INFO - Loop took: 0.453813 seconds
[2016-12-18 00:15:06,477] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:06,477] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:51.477330
[2016-12-18 00:15:11,021] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:11,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:11,099] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,100] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,102] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:11,105] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:11,159] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,160] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,160] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,162] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,165] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,206] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,208] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,209] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,211] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,211] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:11,214] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,262] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,264] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,267] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,309] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,311] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,314] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:11,316] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,318] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,321] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,371] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,372] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,375] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,397] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,399] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,401] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:11,446] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:11,448] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:11,451] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:11,475] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:11,475] {jobs.py:744} INFO - Loop took: 0.460182 seconds
[2016-12-18 00:15:11,485] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:11,486] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:12:56.485998
[2016-12-18 00:15:16,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:16,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:16,123] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,125] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,127] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:16,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,133] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:16,180] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,189] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,195] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,234] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,238] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:16,243] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,245] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,248] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,304] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,306] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,308] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,359] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,361] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,362] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,365] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:16,367] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,420] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,422] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,425] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,450] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,453] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,456] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:16,505] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:16,507] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:16,510] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:16,538] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:16,538] {jobs.py:744} INFO - Loop took: 0.526617 seconds
[2016-12-18 00:15:16,548] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:16,548] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:01.548491
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:15:22,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:15:22,382] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:15:22,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:15:22,591] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:15:22,591] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:15:22,719] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:15:22,742] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:22,751] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:22,791] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,792] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,793] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,794] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,796] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:22,796] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:22,844] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,846] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,846] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,848] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,849] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:22,855] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:22,895] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,897] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,897] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,899] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,899] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:22,901] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:22,942] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,944] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,947] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:22,989] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:22,991] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:22,994] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:23,006] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:23,008] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:23,011] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:23,040] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:23,042] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:23,045] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:23,092] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:23,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:23,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:23,146] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:23,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:23,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:23,179] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:23,179] {jobs.py:744} INFO - Loop took: 0.443431 seconds
[2016-12-18 00:15:23,189] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:23,189] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:08.189630
[2016-12-18 00:15:27,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:27,028] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:27,083] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,083] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,085] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,085] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,088] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:27,088] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:27,146] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,148] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,151] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:27,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:27,200] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:27,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:27,254] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,256] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:27,307] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,309] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,311] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:27,323] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:27,325] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:27,328] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:27,447] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:27,447] {jobs.py:744} INFO - Loop took: 0.433547 seconds
[2016-12-18 00:15:27,462] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:27,462] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:12.462611
[2016-12-18 00:15:32,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:32,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:32,079] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,081] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,084] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,084] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:32,086] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,089] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:32,134] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,136] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,140] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,140] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,142] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,189] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,191] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,194] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:32,194] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,196] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,200] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,248] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,254] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,303] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,305] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,308] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:32,361] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,363] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,366] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,518] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,520] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,523] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,612] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,614] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,618] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:32,666] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:32,668] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:32,671] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:32,699] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:32,699] {jobs.py:744} INFO - Loop took: 0.687573 seconds
[2016-12-18 00:15:32,709] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:32,709] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:17.709941
[2016-12-18 00:15:37,015] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:37,024] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:37,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:37,099] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:37,142] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,144] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,147] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,156] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,158] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,161] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,193] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,195] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,198] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:37,208] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,210] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,213] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,260] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,262] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,265] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,315] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,317] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,318] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,320] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:37,323] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,372] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,374] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,377] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,407] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,409] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,413] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:37,462] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:37,464] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:37,467] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:37,495] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:37,495] {jobs.py:744} INFO - Loop took: 0.485812 seconds
[2016-12-18 00:15:37,505] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:37,505] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:22.505858
[2016-12-18 00:15:42,020] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:42,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:42,104] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,106] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,108] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,109] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:42,111] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,113] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:42,159] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,163] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,169] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,171] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,174] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,210] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,212] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,215] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:42,223] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,225] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,228] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,275] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,277] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,280] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,329] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,332] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,335] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:42,343] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,345] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,348] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,389] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,391] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,394] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,445] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,447] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,450] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:42,502] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:42,505] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:42,508] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:42,536] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:42,537] {jobs.py:744} INFO - Loop took: 0.522155 seconds
[2016-12-18 00:15:42,548] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:42,548] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:27.548460
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:15:48,320] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:15:48,381] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:15:48,399] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:15:48,608] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:15:48,608] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:15:48,735] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:15:48,758] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:48,767] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:48,808] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,811] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,813] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,814] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:48,815] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,818] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:48,866] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,868] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,869] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,871] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,871] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:48,874] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:48,918] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,920] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,921] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,923] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:48,923] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,926] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:48,973] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:48,975] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:48,977] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:49,024] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:49,026] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:49,029] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:49,041] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:49,043] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:49,046] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:49,076] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:49,078] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:49,081] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:49,120] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:49,122] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:49,125] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:49,169] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:49,171] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:49,175] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:49,202] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:49,203] {jobs.py:744} INFO - Loop took: 0.451055 seconds
[2016-12-18 00:15:49,213] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:49,213] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:34.213326
[2016-12-18 00:15:53,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:53,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:53,088] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,088] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,090] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,091] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,093] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:53,094] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:53,147] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,152] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:53,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:53,200] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,202] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:53,205] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:53,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:53,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:53,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:53,383] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:53,383] {jobs.py:744} INFO - Loop took: 0.37249 seconds
[2016-12-18 00:15:53,393] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:53,393] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:38.393917
[2016-12-18 00:15:58,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:15:58,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:15:58,094] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,094] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,096] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:58,099] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:58,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,161] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,163] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,166] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,166] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,213] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,215] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,217] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,218] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:58,219] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,222] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,270] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,272] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,275] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,326] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:58,331] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,332] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,336] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,376] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,378] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,381] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,421] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,423] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,426] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:15:58,480] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:15:58,481] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:15:58,484] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:15:58,508] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:15:58,508] {jobs.py:744} INFO - Loop took: 0.49668 seconds
[2016-12-18 00:15:58,518] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:15:58,519] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:43.519031
[2016-12-18 00:16:03,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:03,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:03,081] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,083] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,086] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,087] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:03,088] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,091] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:03,143] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,146] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,199] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,201] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,204] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:03,206] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,256] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,258] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,261] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,309] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,311] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,315] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:03,330] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,332] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,335] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,367] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,369] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,372] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,425] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,427] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,430] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:03,479] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:03,481] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:03,485] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:03,518] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:03,518] {jobs.py:744} INFO - Loop took: 0.50708 seconds
[2016-12-18 00:16:03,530] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:03,530] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:48.530370
[2016-12-18 00:16:08,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:08,027] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:08,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:08,133] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,135] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,138] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:08,187] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,189] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,192] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,192] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,194] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,197] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,246] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,247] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,248] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,250] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:08,253] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,299] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,301] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,303] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,356] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,358] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,361] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:08,369] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,371] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,374] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,417] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,419] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,422] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,461] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,463] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,466] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:08,515] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:08,516] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:08,519] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:08,544] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:08,544] {jobs.py:744} INFO - Loop took: 0.533103 seconds
[2016-12-18 00:16:08,554] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:08,554] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:13:53.554967
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:16:14,326] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:16:14,387] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:16:14,404] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:16:14,612] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:16:14,613] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:16:14,740] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:16:14,763] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:14,772] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:14,814] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,815] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,816] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,817] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,819] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:14,820] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:14,878] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,880] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,881] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,883] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:14,883] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,886] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:14,929] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,932] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,932] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,934] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,935] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:14,937] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:14,988] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:14,990] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:14,993] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:15,039] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:15,041] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:15,044] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:15,060] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:15,062] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:15,065] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:15,096] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:15,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:15,101] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:15,147] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:15,149] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:15,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:15,204] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:15,206] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:15,209] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:15,237] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:15,237] {jobs.py:744} INFO - Loop took: 0.480471 seconds
[2016-12-18 00:16:15,247] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:15,247] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:00.247248
[2016-12-18 00:16:19,018] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:19,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:19,127] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,132] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:19,133] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,136] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:19,181] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,183] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,186] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,186] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:19,188] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,190] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:19,231] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,234] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,234] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,236] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,236] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:19,239] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:19,284] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:19,286] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:19,289] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:19,414] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:19,414] {jobs.py:744} INFO - Loop took: 0.402493 seconds
[2016-12-18 00:16:19,424] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:19,424] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:04.424968
[2016-12-18 00:16:24,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:24,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:24,092] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,092] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:24,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:24,148] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,150] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,150] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,152] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,153] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,156] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,199] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,201] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,202] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:24,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,252] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,254] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,257] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,305] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,309] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:24,311] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,313] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,317] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,361] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,363] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,366] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,400] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,402] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,405] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:24,453] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:24,455] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:24,458] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:24,485] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:24,486] {jobs.py:744} INFO - Loop took: 0.474028 seconds
[2016-12-18 00:16:24,495] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:24,496] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:09.496204
[2016-12-18 00:16:29,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:29,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:29,068] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,070] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,072] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:29,093] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,094] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,097] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:29,121] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,123] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,126] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,145] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,170] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,172] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,174] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,202] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:29,218] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,220] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,223] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,277] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,279] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,282] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:29,317] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,319] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,323] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,335] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,337] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,340] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,406] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,408] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,411] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:29,458] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:29,460] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:29,463] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:29,498] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:29,499] {jobs.py:744} INFO - Loop took: 0.486877 seconds
[2016-12-18 00:16:29,508] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:29,509] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:14.509096
[2016-12-18 00:16:34,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:34,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:34,072] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,074] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,075] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,076] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:34,078] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,080] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:34,124] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,126] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,131] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,132] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,135] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,169] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,171] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,174] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:34,176] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,178] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,181] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,225] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,227] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,229] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,275] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,276] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,280] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,280] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:34,282] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,285] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,333] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,335] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,339] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,370] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,372] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,375] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:34,422] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:34,424] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:34,428] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:34,455] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:34,455] {jobs.py:744} INFO - Loop took: 0.443953 seconds
[2016-12-18 00:16:34,465] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:34,466] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:19.466083
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:16:40,319] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:16:40,380] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:16:40,397] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:16:40,606] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:16:40,606] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:16:40,733] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:16:40,756] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:40,765] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:40,805] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,806] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,807] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,808] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,809] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:40,810] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:40,858] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,859] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,860] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,861] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,863] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:40,864] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:40,906] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,907] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,907] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,909] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,910] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:40,912] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:40,959] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:40,961] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:40,964] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:41,008] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:41,010] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:41,013] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:41,014] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:41,015] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:41,018] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:41,068] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:41,070] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:41,073] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:41,099] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:41,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:41,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:41,152] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:41,154] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:41,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:41,184] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:41,185] {jobs.py:744} INFO - Loop took: 0.434987 seconds
[2016-12-18 00:16:41,194] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:41,195] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:26.195077
[2016-12-18 00:16:45,022] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:45,031] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:45,095] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,098] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:45,106] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,108] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,111] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:45,149] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,151] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,154] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:45,159] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,164] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:45,196] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,197] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,200] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:45,214] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,216] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,218] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:45,263] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,265] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,269] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:45,321] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:45,323] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:45,326] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:45,380] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:45,380] {jobs.py:744} INFO - Loop took: 0.364235 seconds
[2016-12-18 00:16:45,390] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:45,390] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:30.390437
[2016-12-18 00:16:50,029] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:50,038] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:50,144] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,145] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,146] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,147] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,149] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:50,150] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:50,196] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,198] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,198] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,200] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,201] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,204] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,248] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,250] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,253] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,253] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:50,255] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,258] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,305] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,307] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,310] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,355] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,357] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,361] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:50,368] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,370] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,374] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,410] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,412] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,415] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,461] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,463] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,466] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:50,515] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:50,517] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:50,520] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:50,548] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:50,548] {jobs.py:744} INFO - Loop took: 0.52455 seconds
[2016-12-18 00:16:50,558] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:50,558] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:35.558297
[2016-12-18 00:16:55,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:16:55,026] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:16:55,098] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,099] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,100] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,101] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,103] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:55,104] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:55,159] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,160] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,161] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,162] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,164] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,164] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,210] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,212] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,213] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,215] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,217] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,218] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:55,265] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,267] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,269] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,318] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,320] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,322] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:55,338] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,340] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,344] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,375] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,377] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,380] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,433] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,435] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,438] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:16:55,488] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:16:55,490] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:16:55,494] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:16:55,521] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:16:55,522] {jobs.py:744} INFO - Loop took: 0.510308 seconds
[2016-12-18 00:16:55,532] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:16:55,532] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:40.532498
[2016-12-18 00:17:00,019] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:17:00,029] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:17:00,097] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,099] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,101] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,102] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:00,103] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,105] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:00,151] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,155] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,155] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,156] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,159] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,197] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,199] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,200] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,201] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:00,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,205] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,251] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,253] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,256] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,304] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,306] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,310] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:00,330] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,332] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,335] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,362] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,422] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,424] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,428] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:00,477] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:00,479] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:00,482] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:00,519] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:17:00,520] {jobs.py:744} INFO - Loop took: 0.506484 seconds
[2016-12-18 00:17:00,530] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:17:00,530] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:45.530825
Scheduler exited with code 0. Respawning..
Running
[2016-12-18 00:17:06,321] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-12-18 00:17:06,383] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-12-18 00:17:06,400] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-12-18 00:17:06,641] {jobs.py:680} INFO - Starting the scheduler
[2016-12-18 00:17:06,642] {models.py:154} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2016-12-18 00:17:06,768] {base_hook.py:53} INFO - Using connection to: bugzilla.cppmil15uwvg.us-west-2.rds.amazonaws.com
[2016-12-18 00:17:06,791] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:17:06,800] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:17:06,840] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,841] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,842] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,843] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,845] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:06,846] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:06,893] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,895] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,895] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,896] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,898] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:06,899] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:06,942] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,944] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,947] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:06,951] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,952] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:06,955] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:06,997] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:06,999] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,002] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:07,046] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:07,047] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,050] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:07,058] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:07,060] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,063] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:07,102] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:07,104] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,107] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:07,143] {models.py:2660} INFO - Checking state for <DagRun android_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:07,145] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,148] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:07,202] {models.py:2660} INFO - Checking state for <DagRun crash_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:07,204] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:07,207] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:07,235] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:17:07,236] {jobs.py:744} INFO - Loop took: 0.450676 seconds
[2016-12-18 00:17:07,245] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:17:07,246] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:52.246151
[2016-12-18 00:17:11,017] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:17:11,025] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:17:11,124] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,124] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,126] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,126] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,128] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:11,129] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:11,179] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,180] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,180] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,182] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,183] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:11,185] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:11,228] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,229] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,230] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,231] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,233] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:11,234] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:11,278] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:11,280] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:11,283] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:11,392] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-12-18 00:17:11,392] {jobs.py:744} INFO - Loop took: 0.381142 seconds
[2016-12-18 00:17:11,403] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-12-18 00:17:11,403] {models.py:311} INFO - Failing jobs without heartbeat after 2016-12-18 00:14:56.403587
[2016-12-18 00:17:16,014] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-12-18 00:17:16,022] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-12-18 00:17:16,073] {models.py:2660} INFO - Checking state for <DagRun bugzilla_dataset @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,075] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,077] {models.py:2660} INFO - Checking state for <DagRun telemetry_aggregates @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,078] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:16,079] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,082] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:16,127] {models.py:2660} INFO - Checking state for <DagRun socorro_import @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,129] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,132] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:16,151] {models.py:2660} INFO - Checking state for <DagRun mobile_clients @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,153] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,157] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:16,176] {models.py:2660} INFO - Checking state for <DagRun longitudinal @ 2016-12-11 00:00:00: scheduled__2016-12-11T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,178] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,181] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:16,201] {models.py:2660} INFO - Checking state for <DagRun client_count @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,203] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,206] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:16,254] {models.py:2660} INFO - Checking state for <DagRun android_addons @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,256] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,259] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:16,296] {models.py:2660} INFO - Checking state for <DagRun main_summary @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,297] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,301] {jobs.py:514} INFO - Checking dependencies on 4 tasks instances, minus 1 skippable ones
[2016-12-18 00:17:16,306] {models.py:2660} INFO - Checking state for <DagRun sync_view @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,308] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,311] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
[2016-12-18 00:17:16,362] {models.py:2660} INFO - Checking state for <DagRun android_events @ 2016-12-17 00:00:00: scheduled__2016-12-17T00:00:00, externally triggered: False>
[2016-12-18 00:17:16,364] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-12-18 00:17:16,367] {jobs.py:514} INFO - Checking dependencies on 1 tasks instances, minus 1 skippable ones
View raw

(Sorry about that, but we can’t show files that are this big right now.)

View raw

(Sorry about that, but we can’t show files that are this big right now.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment