gistfile1.txt

[2016-09-22 18:50:23,412] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-09-22 18:50:23,413] {jobs.py:744} INFO - Loop took: 0.278067 seconds
[2016-09-22 18:50:23,422] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-09-22 18:50:23,423] {models.py:311} INFO - Failing jobs without heartbeat after 2016-09-22 18:48:08.423588
[2016-09-22 18:50:28,120] {jobs.py:574} INFO - Prioritizing 0 queued jobs
[2016-09-22 18:50:28,122] {models.py:154} INFO - Filling up the DagBag from /opt/t3/airflow/dags
[2016-09-22 18:50:28,144] {jobs.py:726} INFO - Starting 2 scheduler jobs
[2016-09-22 18:50:28,258] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 12:20:00: scheduled__2016-09-22T12:20:00, externally triggered: False>
[2016-09-22 18:50:28,258] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 13:20:00: scheduled__2016-09-22T13:20:00, externally triggered: False>
[2016-09-22 18:50:28,258] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 14:20:00: scheduled__2016-09-22T14:20:00, externally triggered: False>
[2016-09-22 18:50:28,259] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 15:20:00: scheduled__2016-09-22T15:20:00, externally triggered: False>
[2016-09-22 18:50:28,259] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 16:20:00: scheduled__2016-09-22T16:20:00, externally triggered: False>
[2016-09-22 18:50:28,259] {models.py:2660} INFO - Checking state for <DagRun anr @ 2016-09-22 17:20:00: scheduled__2016-09-22T17:20:00, externally triggered: False>
[2016-09-22 18:50:28,260] {jobs.py:498} INFO - Getting list of tasks to skip for active runs.
[2016-09-22 18:50:28,262] {jobs.py:514} INFO - Checking dependencies on 6 tasks instances, minus 1 skippable ones
[2016-09-22 18:50:28,330] {jobs.py:741} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-09-22 18:50:28,331] {jobs.py:744} INFO - Loop took: 0.214597 seconds
[2016-09-22 18:50:28,341] {models.py:305} INFO - Finding 'running' jobs without a recent heartbeat
[2016-09-22 18:50:28,342] {models.py:311} INFO - Failing jobs without heartbeat after 2016-09-22 18:48:13.342415
Logging into: /data/log/airflow/anr/refresh_models/2016-09-22T12:20:00



(tve)push@prod-db2:PROD::~ $ ps aux | grep scheduler
push     10687  0.0  0.0   6044   476 pts/3    S+   09:54   0:00 tail -f scheduler.log
app      25473  0.5  0.9 449508 75760 ?        Sl   18:49   0:06 /tve/bin/python /tve/bin/airflow scheduler --num_runs 10
app      25484  0.0  0.8 383176 65176 ?        S    18:49   0:00 /tve/bin/python /tve/bin/airflow scheduler --num_runs 10
app      25485  0.0  0.8 383176 65172 ?        S    18:49   0:00 /tve/bin/python /tve/bin/airflow scheduler --num_runs 10
push     26184  0.0  0.0  12944   976 pts/5    S+   19:11   0:00 grep --color=auto scheduler
(tve)push@prod-db2:PROD::~ $ sudo strace -p 25473
strace: Process 25473 attached
futex(0x7f632c700000, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff^Cstrace: Process 25473 detached
 <detached ...>
(tve)push@prod-db2:PROD::~ $ sudo strace -p 25484
strace: Process 25484 attached
wait4(25501, ^Cstrace: Process 25484 detached
 <detached ...>
(tve)push@prod-db2:PROD::~ $ sudo strace -p 25485
strace: Process 25485 attached
wait4(25500, ^Cstrace: Process 25485 detached
 <detached ...>