gistfile1.txt

Verbose logs from DC/OS CLI:

gmann@gmac:~/Downloads/bundle-2018-10-10-1539185632/10.138.0.6_agent|  dcos-1.12 --debug --log-level=DEBUG task exec -it prometheus-0-server__8b619a20-342f-4ecf-8715-8eecc0f8c55c bash
MainThread: 2018-10-15 09:57:35,344 dcos/http.py:_request:163 - Sending HTTP ['get'] to ['https://mwst9.scaletesting.mesosphe.re/mesos/master/state.json']: {'Accept': 'application/json'}
MainThread: 2018-10-15 09:57:35,352 urllib3/connectionpool.py:_new_conn:823 - Starting new HTTPS connection (1): mwst9.scaletesting.mesosphe.re:443
send: b'GET /mesos/master/state.json HTTP/1.1\r\nHost: mwst9.scaletesting.mesosphe.re\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/json\r\nConnection: keep-alive\r\nUser-Agent: python-requests/2.19.1\r\nAuthorization: token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJ1aWQiOiJkY29zLWRldiIsImV4cCI6MTUzOTYyNjIzOH0.haWtC3vb4KrUnovam-MMJgWmOKj0HruaZ5rkzBQ0asAITpkEkSVpYDDVdZhAq1ZGoJt8bb8mWgko6s4XBWqNCZQL8FHZo8fxDQXUN1g681H-dUhwCCiNDKUOVNsgjWCriBIgcPqdTSgi9Zp0iri21I7yFWKDWmdIaT6_V3MBaldmNngJ3zIB7woSyb-6vHbcc6OLE3uTYVnfAaW1uqWkz83gNY5rJRzuHGAwanh5Jq9t452Ziauhou8DEaCeJmYjoIeVOIrrmnevClHQu-PKrgkAaJ_pzcmXMWCnfG_zyA0AS7tg77PFjnj-1i2MwD0Wy0Q894IcssJQR_Fv8w8ymQ\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
MainThread: 2018-10-15 09:57:41,113 urllib3/connectionpool.py:_make_request:393 - https://mwst9.scaletesting.mesosphe.re:443 "GET /mesos/master/state.json HTTP/1.1" 200 3986842
MainThread: 2018-10-15 09:57:42,056 dcos/http.py:_request:193 - Received HTTP response [200]: {'Content-Encoding': 'gzip', 'Server': 'openresty', 'Content-Type': 'application/json', 'Content-Length': '3986842', 'Connection': 'keep-alive', 'Date': 'Mon, 15 Oct 2018 16:57:41 GMT'}
MainThread: 2018-10-15 09:57:42,056 dcos/util.py:timer:609 - duration: dcos.http._request: 6.71s
Thread-4: 2018-10-15 09:57:43,037 dcos/http.py:_request:163 - Sending HTTP ['post'] to ['https://mwst9.scaletesting.mesosphe.re/slave/113720af-edaf-4ff1-b0da-b3b381a0a32f-S140/api/v1']: {'Message-Accept': 'application/json', 'Content-Type': 'application/json', 'Accept': 'application/recordio'}
                                                        Thread-4: 2018-10-15 09:57:43,040 urllib3/connectionpool.py:_new_conn:823 - Starting new HTTPS connection (1): mwst9.scaletesting.mesosphe.re:443
                                                                                                                                                                                                         header: Date header: Content-Type header: Content-Length header: Connection header: Server header: Content-Encoding send: b'POST /slave/113720af-edaf-4ff1-b0da-b3b381a0a32f-S140/api/v1 HTTP/1.1\r\nHost: mwst9.scaletesting.mesosphe.re\r\nAccept-Encoding: gzip, deflate\r\nAccept: application/recordio\r\nConnection: keep-alive\r\nUser-Agent: python-requests/2.19.1\r\nMessage-Accept: application/json\r\nContent-Type: application/json\r\nContent-Length: 386\r\nAuthorization: token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJ1aWQiOiJkY29zLWRldiIsImV4cCI6MTUzOTYyNjIzOH0.haWtC3vb4KrUnovam-MMJgWmOKj0HruaZ5rkzBQ0asAITpkEkSVpYDDVdZhAq1ZGoJt8bb8mWgko6s4XBWqNCZQL8FHZo8fxDQXUN1g681H-dUhwCCiNDKUOVNsgjWCriBIgcPqdTSgi9Zp0iri21I7yFWKDWmdIaT6_V3MBaldmNngJ3zIB7woSyb-6vHbcc6OLE3uTYVnfAaW1uqWkz83gNY5rJRzuHGAwanh5Jq9t452Ziauhou8DEaCeJmYjoIeVOIrrmnevClHQu-PKrgkAaJ_pzcmXMWCnfG_zyA0AS7tg77PFjnj-1i2MwD0Wy0Q894IcssJQR_Fv8w8ymQ\r\n\r\n'
                                                                                                                                                                      send: b'{"launch_nested_container_session": {"container": {"tty_info": {}, "type": "MESOS"}, "container_id": {"value": "d226d512-9285-498f-8198-871dda3f6c5b", "parent": {"value": "acc36578-8ff9-418c-8635-9ecedb03d8e0", "parent": {"value": "1dc29303-aa4a-4690-8cc4-3a5e1c99e11c"}}}, "command": {"arguments": ["bash"], "value": "bash", "shell": false}}, "type": "LAUNCH_NESTED_CONTAINER_SESSION"}'
                                                                                     reply: 'HTTP/1.1 500 Internal Server Error\r\n'
                                                                                                                                    Thread-4: 2018-10-15 09:58:26,462 urllib3/connectionpool.py:_make_request:393 - https://mwst9.scaletesting.mesosphe.re:443 "POST /slave/113720af-edaf-4ff1-b0da-b3b381a0a32f-S140/api/v1 HTTP/1.1" 500 220
                                                                                                Thread-4: 2018-10-15 09:58:26,463 dcos/http.py:_request:193 - Received HTTP response [500]: {'Server': 'openresty', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '220', 'Connection': 'keep-alive', 'Date': 'Mon, 15 Oct 2018 16:58:26 GMT'}
                                                                                                                Thread-4: 2018-10-15 09:58:26,463 dcos/util.py:timer:609 - duration: dcos.http._request: 43.43s
                                                                                                                                                                                                               Error while fetching [https://mwst9.scaletesting.mesosphe.re/slave/113720af-edaf-4ff1-b0da-b3b381a0a32f-S140/api/v1]: HTTP 500: "Internal Server Error".
header: Date header: Content-Type header: Content-Length header: Connection header: Server %


From the Mesos agent logs at that time; seems possible that the 500 is due to failed authorization:
I1015 16:57:52.418298 24027 http.cpp:3348] Processing LAUNCH_NESTED_CONTAINER_SESSION call for container '1dc29303-aa4a-4690-8cc4-3a5e1c99e11c.acc36578-8ff9-418c-8635-9ecedb03d8e0.d226d512-9285-498f-8198-871dda3f6c5b'
W1015 16:57:53.292436 24037 bouncer_utils.hpp:209] Failed to execute retriable action after multiple attempts. Giving up. Last message was: 'Failed to determine IP of domain 'master.mesos': Name or service not known'