Created
March 3, 2019 20:36
-
-
Save edraizen/64580011eed6c23d8a2448357876fa53 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| DEBUG:toil.batchSystems.mesos.batchSystem:Preparing to launch Mesos task 0 with 1.00 cores, 2048.00 MiB memory, and 2048.00 MiB disk using offer 5b6aaf7d-99c1-410a-85f3-dc9602313361-O20 ... | |
| DEBUG:toil.batchSystems.mesos.batchSystem:Launched Mesos task 0. | |
| WARNING:toil.batchSystems.mesos.batchSystem:Executor 'toil-28072' reported lost with status '32512'. | |
| WARNING:toil.batchSystems.mesos.batchSystem:Handling failure of executor 'toil-28072' on agent '5b6aaf7d-99c1-410a-85f3-dc9602313361-S2'. | |
| DEBUG:toil.batchSystems.mesos.batchSystem:Available files: dict_keys(['/slave/log', '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace', '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f']) | |
| WARNING:toil.batchSystems.mesos.batchSystem:Attempting to retrieve executor error log: http://172.31.36.66:5051/files/download?path=%2Fvar%2Flib%2Fmesos%2Fslaves%2F5b6aaf7d-99c1-410a-85f3-dc9602313361-S2%2Fframeworks%2F5b6aaf7d-99c1-410a-85f3-dc9602313361-0033%2Fexecutors%2Ftoil-28072%2Fruns%2F807e0102-ecc0-406c-b9b8-f60a845ec65f%2Fstderr | |
| WARNING:toil.batchSystems.mesos.batchSystem:Executor: b'sh: 1: _toil_mesos_executor: not found' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Attempting to retrieve agent log: http://172.31.36.66:5051/files/download?path=%2Fslave%2Flog | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Log file created at: 2019/03/03 20:06:12' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Running on machine: ip-172-31-36-66.ec2.internal' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.310715 1 logging.cpp:194] INFO level logging started!' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.438370 1 systemd.cpp:237] systemd version `229` detected' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.439499 1 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.444424 1 main.cpp:434] Starting Mesos agent' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.445698 7 slave.cpp:198] Agent started on 1)@172.31.36.66:5051' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.445739 7 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --attributes="preemptable:True" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="false" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/lib/mesos" --logbufsecs="0" --logging_level="INFO" --master="172.31.41.201:5050" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="false" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446818 7 slave.cpp:519] Agent resources: cpus(*):2; mem(*):6449; disk(*):45020; ports(*):[31000-32000]' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446892 7 slave.cpp:527] Agent attributes: [ preemptable=True ]' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.446925 7 slave.cpp:532] Agent hostname: 172.31.36.66' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:06:12.450480 11 state.cpp:57] Recovering state from '/var/lib/mesos/meta'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.450983 9 status_update_manager.cpp:200] Recovering status update manager' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.451218 9 containerizer.cpp:522] Recovering containerizer' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.451946 8 provisioner.cpp:253] Provisioner recovery complete' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.452318 6 slave.cpp:4782] Finished recovery' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453174 6 status_update_manager.cpp:174] Pausing sending status updates' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453200 9 slave.cpp:895] New master detected at master@172.31.41.201:5050' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453476 9 slave.cpp:916] No credentials provided. Attempting to register without authentication' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.453595 9 slave.cpp:927] Detecting new master' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.572358 10 slave.cpp:1095] Registered with master master@172.31.41.201:5050; given agent ID 5b6aaf7d-99c1-410a-85f3-dc9602313361-S2' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.572865 8 status_update_manager.cpp:181] Resuming sending status updates' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:06:12.573140 10 slave.cpp:1155] Forwarding total oversubscribed resources' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:07:12.463241 11 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125651996422547days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.226449 10 slave.cpp:1495] Got assigned task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.226830 10 slave.cpp:1614] Launching task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.228348 10 paths.cpp:528] Trying to chown '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace' to user 'root'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237267 10 slave.cpp:5674] Launching executor toil-27424 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 with resources in work directory '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237512 6 containerizer.cpp:781] Starting container 'f56c948d-a6b4-4ea5-9639-7518518f6ace' for executor 'toil-27424' of framework '5b6aaf7d-99c1-410a-85f3-dc9602313361-0032'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237512 10 slave.cpp:1840] Queuing task '0' for executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.237677 10 slave.cpp:2218] Asked to shut down framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 by master@172.31.41.201:5050' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.237776 10 slave.cpp:2243] Shutting down framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.237812 10 slave.cpp:4407] Shutting down executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"W0303 20:08:05.237843 10 slave.hpp:768] Unable to send event to executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032: unknown connection type" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.240005 10 launcher.cpp:126] Forked child with pid '19' for container 'f56c948d-a6b4-4ea5-9639-7518518f6ace'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"W0303 20:08:05.241564 10 slave.cpp:4026] Killing executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 because the framework is terminating" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.241880 10 containerizer.cpp:1622] Destroying container 'f56c948d-a6b4-4ea5-9639-7518518f6ace'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.340878 11 containerizer.cpp:1863] Executor for container 'f56c948d-a6b4-4ea5-9639-7518518f6ace' has exited" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342031 12 slave.cpp:4089] Executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 terminated with signal Killed" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342329 12 slave.cpp:4193] Cleaning up executor 'toil-27424' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.342664 12 slave.cpp:4281] Cleaning up framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342828 12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424/runs/f56c948d-a6b4-4ea5-9639-7518518f6ace' for gc 6.99999603579852days in the future" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.342983 12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032/executors/toil-27424' for gc 6.99999603422518days in the future" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:05.343055 12 gc.cpp:55] Scheduling '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' for gc 6.99999603246519days in the future" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:05.343191 12 status_update_manager.cpp:282] Closing status update streams for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:08:10.239328 8 slave.cpp:4448] Framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 seems to have exited. Ignoring shutdown timeout for executor 'toil-27424'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:08:12.464560 13 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:09:05.238062 7 slave.cpp:4499] Framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0032 seems to have exited. Ignoring registration timeout for executor 'toil-27424'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:09:12.465847 9 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:10:12.466612 11 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125644906964490days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:11:12.467156 8 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:12:12.467806 6 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:12.468935 12 slave.cpp:4591] Current disk usage 16.78%. Max allowed age: 5.125643816278634days' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.614944 11 slave.cpp:1495] Got assigned task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.615365 11 slave.cpp:1614] Launching task 0 for framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.616663 11 paths.cpp:528] Trying to chown '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f' to user 'root'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.624841 11 slave.cpp:5674] Launching executor toil-28072 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 with resources in work directory '/var/lib/mesos/slaves/5b6aaf7d-99c1-410a-85f3-dc9602313361-S2/frameworks/5b6aaf7d-99c1-410a-85f3-dc9602313361-0033/executors/toil-28072/runs/807e0102-ecc0-406c-b9b8-f60a845ec65f'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.625484 11 slave.cpp:1840] Queuing task '0' for executor 'toil-28072' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.625499 9 containerizer.cpp:781] Starting container '807e0102-ecc0-406c-b9b8-f60a845ec65f' for executor 'toil-28072' of framework '5b6aaf7d-99c1-410a-85f3-dc9602313361-0033'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.627305 9 launcher.cpp:126] Forked child with pid '22' for container '807e0102-ecc0-406c-b9b8-f60a845ec65f'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.892017 10 containerizer.cpp:1863] Executor for container '807e0102-ecc0-406c-b9b8-f60a845ec65f' has exited" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.892127 10 containerizer.cpp:1622] Destroying container '807e0102-ecc0-406c-b9b8-f60a845ec65f'" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b"I0303 20:13:56.893589 7 slave.cpp:4089] Executor 'toil-28072' of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 exited with status 127" | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.894453 7 slave.cpp:3211] Handling status update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 from @0.0.0.0:0' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'W0303 20:13:56.894974 7 containerizer.cpp:1451] Ignoring update for unknown container: 807e0102-ecc0-406c-b9b8-f60a845ec65f' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.895244 13 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033' | |
| WARNING:toil.batchSystems.mesos.batchSystem:Agent: b'I0303 20:13:56.895579 6 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: b449c399-dc6b-4fc0-8865-a424be91e483) for task 0 of framework 5b6aaf7d-99c1-410a-85f3-dc9602313361-0033 to master@172.31.41.201:5050' | |
| DEBUG:toil.batchSystems.mesos.batchSystem:Job 0 is in state 'TASK_FAILED' due to reason 'REASON_EXECUTOR_TERMINATED'. | |
| WARNING:toil.batchSystems.mesos.batchSystem:Job 0 failed with message 'Executor terminated' due to reason 'REASON_EXECUTOR_TERMINATED' on executor '{'value': 'toil-28072'}' on agent '{'value': '5b6aaf7d-99c1-410a-85f3-dc9602313361-S2'}'. | |
| DEBUG:toil.batchSystems.mesos.batchSystem:Job 0 ended with status 255, took ??? seconds. | |
| WARNING:toil.leader:Job failed with exit value 255: 'start_toil' c4f5769c-ede3-4d06-b7a8-59fb99ecdc29 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment