Skip to content

Instantly share code, notes, and snippets.

@ekesken
Last active December 7, 2015 18:32
Show Gist options
  • Save ekesken/f2edfd65cca8638b0136 to your computer and use it in GitHub Desktop.
Save ekesken/f2edfd65cca8638b0136 to your computer and use it in GitHub Desktop.
marathon task is wrongly assigned to chronos framework after task failure
mesos-slave:
===========
Dec 7 08:58:22 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:22.581887 56133 slave.cpp:1144] Got assigned task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 for framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:22 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:22.582346 56133 slave.cpp:1254] Launching task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 for framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:22 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:22.591339 56133 slave.cpp:4208] Launching executor collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 in work directory '/tmp/mesos/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128/runs/ed5f4f67-464d-4786-9628-cd48732de6b7'
Dec 7 08:58:22 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:22.592087 56133 slave.cpp:1401] Queuing task 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' for executor collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework '20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:22 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:22.596237 56127 docker.cpp:626] Starting container 'ed5f4f67-464d-4786-9628-cd48732de6b7' for task 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' (and executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128') of framework '20151113-112010-100670892-5050-7957-0001'
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.346169 56131 docker.cpp:277] Checkpointing pid 34078 to '/tmp/mesos/meta/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128/runs/ed5f4f67-464d-4786-9628-cd48732de6b7/pids/forked.pid'
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.348412 56131 slave.cpp:3165] Monitoring executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' of framework '20151113-112010-100670892-5050-7957-0001' in container 'ed5f4f67-464d-4786-9628-cd48732de6b7'
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.375448 56131 slave.cpp:2164] Got registration for executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' of framework 20151113-112010-100670892-5050-7957-0001 from executor(1)@172.29.1.12:1651
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.376411 56132 slave.cpp:1555] Sending queued task 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' to executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.381322 56131 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 from executor(1)@172.29.1.12:1651
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.381677 56129 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.381870 56129 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.512739 56129 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to master@172.29.0.5:5050
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.512872 56129 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to executor(1)@172.29.1.12:1651
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.519527 56129 status_update_manager.cpp:389] Received status update acknowledgement (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:25 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:25.519561 56129 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.089156 56130 slave.cpp:2531] Handling status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 from executor(1)@172.29.1.12:1651
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: E1207 08:58:27.089874 56128 slave.cpp:2662] Failed to update resources for container ed5f4f67-464d-4786-9628-cd48732de6b7 of executor collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 running task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 on status update for terminal task, destroying container: Failed to determine cgroup for the 'cpu' subsystem: Failed to read /proc/34074/cgroup: Failed to open file '/proc/34074/cgroup': No such file or directory
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.089959 56128 status_update_manager.cpp:317] Received status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.089978 56128 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.149153 56130 slave.cpp:2776] Forwarding the update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to master@172.29.0.5:5050
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.149317 56130 slave.cpp:2709] Sending acknowledgement for status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to executor(1)@172.29.1.12:1651
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.176398 56134 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:27 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:27.176427 56134 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.162634 56127 slave.cpp:3223] Executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' of framework 20151113-112010-100670892-5050-7957-0001 exited with status 0
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.162680 56127 slave.cpp:3332] Cleaning up executor 'collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.163069 56127 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128/runs/ed5f4f67-464d-4786-9628-cd48732de6b7' for gc 6.99999811536889days in the future
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.163122 56127 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' for gc 6.99999811487407days in the future
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.163143 56127 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128/runs/ed5f4f67-464d-4786-9628-cd48732de6b7' for gc 6.9999981145363days in the future
Dec 7 08:58:28 mesos-slave-node-012 mesos-slave[56099]: I1207 08:58:28.163167 56127 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20151201-192538-83893676-5050-7646-S44/frameworks/20151113-112010-100670892-5050-7957-0001/executors/collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128' for gc 6.99999811425778days in the future
Dec 7 09:03:27 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:27.653187 56130 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from executor(1)@172.29.1.12:1651
Dec 7 09:03:27 mesos-slave-node-012 mesos-slave[56099]: W1207 09:03:27.653282 56130 slave.cpp:2568] Could not find the executor for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:27 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:27.653390 56130 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:27 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:27.653543 56130 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 09:03:27 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:27.653688 56130 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to executor(1)@172.29.1.12:1651
Dec 7 09:03:37 mesos-slave-node-012 mesos-slave[56099]: W1207 09:03:37.654337 56134 status_update_manager.cpp:472] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:37 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:37.654585 56134 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 09:03:57 mesos-slave-node-012 mesos-slave[56099]: W1207 09:03:57.655761 56127 status_update_manager.cpp:472] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:57 mesos-slave-node-012 mesos-slave[56099]: I1207 09:03:57.655872 56127 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 09:04:28 mesos-slave-node-012 mesos-slave[56099]: I1207 09:04:28.093690 56132 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: 60ca68b8-0730-4659-b7b9-4841890496df) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from executor(1)@172.29.1.12:1651
Dec 7 09:04:28 mesos-slave-node-012 mesos-slave[56099]: W1207 09:04:28.093739 56132 slave.cpp:2568] Could not find the executor for status update TASK_RUNNING (UUID: 60ca68b8-0730-4659-b7b9-4841890496df) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:04:28 mesos-slave-node-012 mesos-slave[56099]: I1207 09:04:28.094202 56134 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: 60ca68b8-0730-4659-b7b9-4841890496df) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:04:28 mesos-slave-node-012 mesos-slave[56099]: I1207 09:04:28.094269 56134 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: 60ca68b8-0730-4659-b7b9-4841890496df) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to executor(1)@172.29.1.12:1651
Dec 7 09:04:37 mesos-slave-node-012 mesos-slave[56099]: W1207 09:04:37.656251 56128 status_update_manager.cpp:472] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:04:37 mesos-slave-node-012 mesos-slave[56099]: I1207 09:04:37.656452 56128 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 09:05:57 mesos-slave-node-012 mesos-slave[56099]: W1207 09:05:57.657177 56129 status_update_manager.cpp:472] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:05:57 mesos-slave-node-012 mesos-slave[56099]: I1207 09:05:57.657682 56134 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 09:08:37 mesos-slave-node-012 mesos-slave[56099]: W1207 09:08:37.658298 56133 status_update_manager.cpp:472] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:08:37 mesos-slave-node-012 mesos-slave[56099]: I1207 09:08:37.658445 56133 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
...
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.238075 56128 slave.cpp:1945] Updating framework 20150624-210230-117448108-5050-3678-0001 pid to scheduler-51d2a4e5-4376-45f1-8817-c709e51b542f@172.29.0.7:8939
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.238579 56128 status_update_manager.cpp:178] Resuming sending status updates
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.238600 56128 status_update_manager.cpp:185] Resending status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.238674 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: dc040642-6368-47d0-ab20-399a298fec39) for task ct:1449478800000:0:engine--tr--insurance-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.238715 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: 4d023d0f-52f3-49bd-8bc9-0876aca1bc12) for task ct:1449478800000:0:engine--tr--insurance-tr-corporate--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.238729 56130 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.238750 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: 39fe3edb-1360-4934-81a3-ffed62b4505e) for task ct:1449478800000:0:engine--en--snacks-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.238999 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: 79436a26-e806-4ada-90eb-8e72499106b5) for task ct:1449478800000:0:engine--en--alcohol-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.239039 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: eea8418a-0fad-41fc-a71f-c093b5069ae2) for task ct:1449478800000:0:engine--en--beauty-cosmetics-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.239073 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: e7a97a0b-ec5d-4d18-b2cb-4685e1b1cd2a) for task ct:1449478800000:0:engine--tr--fastfood-tr-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.239105 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: 55e69a8d-1dc2-44e2-98d2-a1a43e9b7f4b) for task ct:1449478800000:0:engine--en--hotels-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: W1207 12:35:25.239158 56128 status_update_manager.cpp:185] Resending status update TASK_FINISHED (UUID: 6aaaa0cd-13eb-4212-8d02-f7203d039ff0) for task ct:1449478800000:0:engine--tr--fastfood-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239167 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: dc040642-6368-47d0-ab20-399a298fec39) for task ct:1449478800000:0:engine--tr--insurance-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239231 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: 4d023d0f-52f3-49bd-8bc9-0876aca1bc12) for task ct:1449478800000:0:engine--tr--insurance-tr-corporate--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239279 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: 39fe3edb-1360-4934-81a3-ffed62b4505e) for task ct:1449478800000:0:engine--en--snacks-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239326 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: 79436a26-e806-4ada-90eb-8e72499106b5) for task ct:1449478800000:0:engine--en--alcohol-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239379 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: eea8418a-0fad-41fc-a71f-c093b5069ae2) for task ct:1449478800000:0:engine--en--beauty-cosmetics-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239464 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: e7a97a0b-ec5d-4d18-b2cb-4685e1b1cd2a) for task ct:1449478800000:0:engine--tr--fastfood-tr-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239512 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: 55e69a8d-1dc2-44e2-98d2-a1a43e9b7f4b) for task ct:1449478800000:0:engine--en--hotels-en-selective--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
Dec 7 12:35:25 mesos-slave-node-012 mesos-slave[56099]: I1207 12:35:25.239557 56130 slave.cpp:2776] Forwarding the update TASK_FINISHED (UUID: 6aaaa0cd-13eb-4212-8d02-f7203d039ff0) for task ct:1449478800000:0:engine--tr--fastfood-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to master@172.29.0.5:5050
mesos-master:
=============
Dec 7 08:58:22 mesos-master-node-001 mesos-master[40898]: I1207 08:58:22.577728 40946 master.hpp:822] Adding task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 with resources cpus(*):0.1; mem(*):200 on slave 20151201-192538-83893676-5050-7646-S44 (172.29.1.12)
Dec 7 08:58:22 mesos-master-node-001 mesos-master[40898]: I1207 08:58:22.577802 40946 master.cpp:2550] Launching task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 (marathon) at scheduler-48b1526a-4692-46bc-a124-9c9d7118c925@172.29.0.6:36187 with resources cpus(*):0.1; mem(*):200 on slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 08:58:25 mesos-master-node-001 mesos-master[40898]: I1207 08:58:25.511479 40947 master.cpp:3300] Status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 08:58:25 mesos-master-node-001 mesos-master[40898]: I1207 08:58:25.511543 40947 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: ec9c6d98-d124-44e8-86c6-e1111b6b0ad9) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:25 mesos-master-node-001 mesos-master[40898]: I1207 08:58:25.511692 40947 master.cpp:4623] Updating the latest state of task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to TASK_RUNNING
Dec 7 08:58:25 mesos-master-node-001 mesos-master[40898]: I1207 08:58:25.516171 40949 master.cpp:2787] Forwarding status update acknowledgement ec9c6d98-d124-44e8-86c6-e1111b6b0ad9 for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 (marathon) at scheduler-48b1526a-4692-46bc-a124-9c9d7118c925@172.29.0.6:36187 to slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 08:58:27 mesos-master-node-001 mesos-master[40898]: I1207 08:58:27.147405 40951 master.cpp:3300] Status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 08:58:27 mesos-master-node-001 mesos-master[40898]: I1207 08:58:27.147466 40951 master.cpp:3341] Forwarding status update TASK_FAILED (UUID: 5b335fab-1722-4270-83a6-b4ec843be47f) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001
Dec 7 08:58:27 mesos-master-node-001 mesos-master[40898]: I1207 08:58:27.147600 40951 master.cpp:4623] Updating the latest state of task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 to TASK_FAILED
Dec 7 08:58:27 mesos-master-node-001 mesos-master[40898]: I1207 08:58:27.172844 40951 master.cpp:4690] Removing task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 with resources cpus(*):0.1; mem(*):200 of framework 20151113-112010-100670892-5050-7957-0001 on slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 08:58:27 mesos-master-node-001 mesos-master[40898]: I1207 08:58:27.172935 40951 master.cpp:2787] Forwarding status update acknowledgement 5b335fab-1722-4270-83a6-b4ec843be47f for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 of framework 20151113-112010-100670892-5050-7957-0001 (marathon) at scheduler-48b1526a-4692-46bc-a124-9c9d7118c925@172.29.0.6:36187 to slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:27 mesos-master-node-001 mesos-master[40898]: I1207 09:03:27.652073 40951 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:27 mesos-master-node-001 mesos-master[40898]: I1207 09:03:27.652119 40951 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:27 mesos-master-node-001 mesos-master[40898]: W1207 09:03:27.652202 40951 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:27 mesos-master-node-001 mesos-master[40898]: I1207 09:03:27.654770 40948 master.cpp:1964] Deactivating framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-7a4396f7-1f68-4f41-901e-805db5de0432@172.29.0.6:11893
Dec 7 09:03:27 mesos-master-node-001 mesos-master[40898]: I1207 09:03:27.654989 40948 hierarchical.hpp:400] Deactivated framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:37 mesos-master-node-001 mesos-master[40898]: I1207 09:03:37.653213 40952 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:37 mesos-master-node-001 mesos-master[40898]: I1207 09:03:37.653290 40952 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:37 mesos-master-node-001 mesos-master[40898]: W1207 09:03:37.653404 40952 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:57 mesos-master-node-001 mesos-master[40898]: I1207 09:03:57.654477 40946 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:03:57 mesos-master-node-001 mesos-master[40898]: I1207 09:03:57.654520 40946 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:03:57 mesos-master-node-001 mesos-master[40898]: W1207 09:03:57.654661 40946 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:04:37 mesos-master-node-001 mesos-master[40898]: I1207 09:04:37.654958 40946 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:04:37 mesos-master-node-001 mesos-master[40898]: I1207 09:04:37.655028 40946 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:04:37 mesos-master-node-001 mesos-master[40898]: W1207 09:04:37.655144 40946 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:05:57 mesos-master-node-001 mesos-master[40898]: I1207 09:05:57.656364 40948 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:05:57 mesos-master-node-001 mesos-master[40898]: I1207 09:05:57.656440 40948 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:05:57 mesos-master-node-001 mesos-master[40898]: W1207 09:05:57.656566 40948 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:08:37 mesos-master-node-001 mesos-master[40898]: I1207 09:08:37.657284 40952 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 09:08:37 mesos-master-node-001 mesos-master[40898]: I1207 09:08:37.657356 40952 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 09:08:37 mesos-master-node-001 mesos-master[40898]: W1207 09:08:37.657465 40952 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
...
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.228590 40949 master.cpp:1711] Received re-registration request from framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-51d2a4e5-4376-45f1-8817-c709e51b542f@172.29.0.7:8939
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.228696 40949 master.cpp:1764] Re-registering framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-51d2a4e5-4376-45f1-8817-c709e51b542f@172.29.0.7:8939
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.228710 40949 master.cpp:1783] Updating info for framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.228899 40949 master.cpp:1798] Framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-7a4396f7-1f68-4f41-901e-805db5de0432@172.29.0.6:11893 failed over
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.230020 40946 hierarchical.hpp:370] Activated framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.235729 40949 master.cpp:3300] Status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.236589 40949 master.cpp:3341] Forwarding status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: W1207 12:35:25.237005 40949 master.cpp:3308] Could not lookup task for status update TASK_RUNNING (UUID: 81aee6b0-2b9d-470a-a543-f14f7cae699b) for task collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 in health state unhealthy of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.237279 40949 master.cpp:3300] Status update TASK_FINISHED (UUID: dc040642-6368-47d0-ab20-399a298fec39) for task ct:1449478800000:0:engine--tr--insurance-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
...
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.242117 40949 master.cpp:3300] Status update TASK_FINISHED (UUID: 6aaaa0cd-13eb-4212-8d02-f7203d039ff0) for task ct:1449478800000:0:engine--tr--fastfood-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 from slave 20151201-192538-83893676-5050-7646-S44 at slave(1)@172.29.1.12:5051 (172.29.1.12)
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.242218 40949 master.cpp:3341] Forwarding status update TASK_FINISHED (UUID: 6aaaa0cd-13eb-4212-8d02-f7203d039ff0) for task ct:1449478800000:0:engine--tr--fastfood-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.242372 40949 master.cpp:4623] Updating the latest state of task ct:1449478800000:0:engine--tr--fastfood-tr-primary--ebv--compare-posts-with-collector: of framework 20150624-210230-117448108-5050-3678-0001 to TASK_FINISHED
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.242496 40949 master.cpp:2611] Reviving offers for framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-51d2a4e5-4376-45f1-8817-c709e51b542f@172.29.0.7:8939
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.242796 40949 http.cpp:516] HTTP request for '/master/state.json'
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.247189 40946 hierarchical.hpp:717] Removed filters for framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.456651 40949 master.cpp:1964] Deactivating framework 20150624-210230-117448108-5050-3678-0001 (chronos-2.4.0) at scheduler-51d2a4e5-4376-45f1-8817-c709e51b542f@172.29.0.7:8939
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.456926 40949 http.cpp:516] HTTP request for '/master/state.json'
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: I1207 12:35:25.457132 40945 hierarchical.hpp:400] Deactivated framework 20150624-210230-117448108-5050-3678-0001
Dec 7 12:35:25 mesos-master-node-001 mesos-master[40898]: W1207 12:35:25.668455 40949 master.cpp:3643] Master returning resources offered to framework 20150624-210230-117448108-5050-3678-0001 because the framework has terminated or is inactive
chronos:
========
no clue at chronos logs between Dec 7 08:58:22 - Dec 7 09:08:37
when we try to restart chronos-master, we got these logs:
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: Exception in thread "Thread-62550" scala.MatchError: collector_tr_insurance_ebv_facebookscraper.ab3ddc6b-9cc0-11e5-8f21-0242ec411128 (of class java.lang.String)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at org.apache.mesos.chronos.scheduler.jobs.TaskUtils$.parseTaskId(TaskUtils.scala:132)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at org.apache.mesos.chronos.scheduler.mesos.MesosJobFramework.statusUpdate(MesosJobFramework.scala:215)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at java.lang.reflect.Method.invoke(Method.java:497)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: #011at com.sun.proxy.$Proxy31.statusUpdate(Unknown Source)
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: I1207 12:35:25.238782 15035 sched.cpp:1623] Asked to abort the driver
Dec 7 12:35:25 mesos-master-node-003 chronos[15003]: I1207 12:35:25.238836 15035 sched.cpp:856] Aborting framework '20150624-210230-117448108-5050-3678-0001'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment