To get a snapshot of the 'world view' we use gen-worldview.sh:
$ curl -O https://gist.githubusercontent.com/mhausenblas/44860b0a7e8688d1822ebaa02522656b/raw/4e3d43c18aabe3885b504a09caecb7c37abf5d79/gen-worldview.sh
$ chmod 755 gen-worldview.sh
$ ./gen-worldview.sh
$ cat 14*
Executing gen-worldview.sh on agent ip-10-0-3-43 yields worldview-agent.txt.
When we launch legacy-app.json as a Marathon app the following is happening on agent ip-10-0-3-43:
A Mesos sandbox /var/lib/mesos/slave/slaves/6522d1b9-0806-41c6-a0fb-b82f20c3c038-S1/frameworks/6522d1b9-0806-41c6-a0fb-b82f20c3c038-0000/executors/legacy-app.e7d94ca7-8e34-11e6-985c-e2bc47a71b50/runs/170661b0-ac21-41ba-a9b5-5460ca1b14d4 was created with the following content:
core@ip-10-0-3-43 $ ls -al
total 56
drwxr-xr-x. 2 root root 4096 Oct 9 15:27 .
drwxr-xr-x. 3 root root 4096 Oct 9 15:27 ..
-rw-r--r--. 1 root root 13616 Oct 9 15:27 1476026856
-rwxr-xr-x. 1 root root 934 Oct 9 15:27 gen-worldview.sh
-rw-r--r--. 1 root root 485 Oct 9 15:27 stderr
-rw-r--r--. 1 root root 255 Oct 9 15:27 stderr.logrotate.conf
-rw-r--r--. 1 root root 15187 Oct 9 15:27 stdout
-rw-r--r--. 1 root root 255 Oct 9 15:27 stdout.logrotate.conf
With stdout containing worldview-legacy-app.txt.
Common tools such as free are not cgroups aware, so we do systemd-cgtop on agent ip-10-0-3-43:
Control Group Tasks %CPU Memory Input/s Output/s
/ - 3.3 2.4G - -
/system.slice 420 1.5 2.4G - -
/user.slice 4 0.9 3.7M - -
/mesos - 0.8 59.3M - -
/system.slice/dcos-minuteman.service 90 0.8 130.0M - -
/system.slice/dcos-navstar.service 90 0.7 85.7M - -
/mesos/slave - 0.6 53.7M - -
/mesos/170661b0-ac21-41ba-a9b5-5460ca1b14d4 - 0.2 4.7M - -
/system.slice/dcos-pkgpanda-api.service 2 0.0 28.7M - -
/system.slice/dcos-spartan.service 126 0.0 97.5M - -
/system.slice/docker.service 15 0.0 248.7M - -
/init.scope 1 - 10.0M - -
/init.scope/system.slice - - 3.3M - -
/system.slice/dbus.service 2 - 2.1M - -
/system.slice/dcos-3dt.service 8 - 10.5M - -
/system.slice/dcos-adminrouter-agent.service 2 - 3.2M - -
/system.slice/dcos-epmd.service 1 - 244.0K - -
/system.slice/dcos-mesos-slave.service 60 - 7.0M - -
/system.slice/dcos-rexray.service 9 - 7.2M - -
/system.slice/polkit.service 5 - 10.0M - -
/system.slice/proc-sys-fs-binfmt_misc.mount - - 80.0K - -
/system.slice/system-getty.slice 1 - - - -
/system.slice/system-getty.slice/getty@tty1.service 1 - - - -
/system.slice/system-serial\x2dgetty.slice 1 - - - -
/system.slice/system-serial\x2dgetty.slice/serial-getty@ttyS0.service 1 - - - -
/system.slice/system-sshd.slice - - 2.6M - -
/system.slice/system-system\x2dcloudinit.slice - - 6.9M - -
/system.slice/systemd-journald.service 1 - 124.0M - -
/system.slice/systemd-logind.service 1 - 1.5M - -
/system.slice/systemd-networkd.service 1 - 1.9M - -
/system.slice/systemd-timesyncd.service 2 - 868.0K - -
/system.slice/systemd-udevd.service 1 - 17.1M - -
/user.slice/user-500.slice 4 - - - -
/user.slice/user-500.slice/session-2.scope 4 - - - -
In /mesos/170661b0-ac21-41ba-a9b5-5460ca1b14d4 we identify the cgroup (task ID, from sandbox). The default path to the cgroups hierarchy root in DC/OS is /sys/fs/cgroup (unchanged from vanilla Apache Mesos). This leads us to the directories /sys/fs/cgroup/$CGRUOP/mesos/170661b0-ac21-41ba-a9b5-5460ca1b14d4, for example, to check on the memory usage:
core@ip-10-0-3-43 $ cd /sys/fs/cgroup/memory/mesos/170661b0-ac21-41ba-a9b5-5460ca1b14d4
core@ip-10-0-3-43 $ cat cat memory.max_usage_in_bytes
6651904
Note that memory.max_usage_in_bytes shows the maximum memory usage recorded.
When we launch type-docker.json as a Marathon app the following is happening on agent ip-10-0-3-43:
core@ip-10-0-3-43 $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0fb8849a255d alpine:3.4 "/bin/sh -c 'apk --no" 6 minutes ago Up 6 minutes mesos-6522d1b9-0806-41c6-a0fb-b82f20c3c038-S1.22c34e3b-222e-4af6-b2a5-ad1056743f3b
Marathon used Mesos to shell out to the Docker engine to run the specified container with a container ID of 0fb8849a255d....
Further, a Mesos sandbox /var/lib/mesos/slave/slaves/6522d1b9-0806-41c6-a0fb-b82f20c3c038-S1/frameworks/6522d1b9-0806-41c6-a0fb-b82f20c3c038-0000/executors/type-docker.fea40dd6-8e2f-11e6-985c-e2bc47a71b50/runs/22c34e3b-222e-4af6-b2a5-ad1056743f3b was created with a stdout containing worldview-type-docker.txt.
When we do systemd-cgtop on agent ip-10-0-3-43:
Control Group Tasks %CPU Memory Input/s Output/s
/ - 6.2 2.4G - -
/system.slice 399 4.2 2.4G - -
/system.slice/dcos-navstar.service 90 2.8 87.7M - -
/system.slice/dcos-minuteman.service 90 1.1 142.2M - -
/mesos - 1.0 50.6M - -
/mesos/slave - 1.0 49.8M - -
/user.slice 4 0.9 3.4M - -
/system.slice/dcos-spartan.service 126 0.2 97.3M - -
/system.slice/dcos-pkgpanda-api.service 2 0.0 28.7M - -
/system.slice/docker.service 14 0.0 248.7M - -
/system.slice/systemd-journald.service 1 0.0 122.9M - -
/init.scope 1 - 10.3M - -
/init.scope/system.slice - - 3.3M - -
/system.slice/dbus.service 2 - 2.1M - -
/system.slice/dcos-3dt.service 8 - 10.5M - -
/system.slice/dcos-adminrouter-agent.service 2 - 3.2M - -
/system.slice/dcos-epmd.service 1 - 244.0K - -
/system.slice/dcos-mesos-slave.service 40 - 7.0M - -
/system.slice/dcos-rexray.service 9 - 7.2M - -
/system.slice/docker-0fb8849a...f3f5a80186da9e7866e5a6d48c05ff2bf7009601e5f3e7fb403.scope 2 - 20.5M - -
/system.slice/polkit.service 5 - 10.0M - -
/system.slice/proc-sys-fs-binfmt_misc.mount - - 80.0K - -
/system.slice/system-getty.slice 1 - - - -
/system.slice/system-getty.slice/getty@tty1.service 1 - - - -
/system.slice/system-serial\x2dgetty.slice 1 - - - -
/system.slice/system-serial\x2dgetty.slice/serial-getty@ttyS0.service 1 - - - -
/system.slice/system-sshd.slice - - 2.6M - -
/system.slice/system-system\x2dcloudinit.slice - - 6.9M - -
/system.slice/systemd-logind.service 1 - 1.5M - -
/system.slice/systemd-networkd.service 1 - 1.9M - -
/system.slice/systemd-timesyncd.service 2 - 868.0K - -
/system.slice/systemd-udevd.service 1 - 17.1M - -
/user.slice/user-500.slice 4 - - - -
/user.slice/user-500.slice/session-2.scope 4 - - - -
We can now change to directory /sys/fs/cgroup/cpu/system.slice/docker-0fb8849a255dcf3f5a80186da9e7866e5a6d48c05ff2bf7009601e5f3e7fb403.scope (the actual directory we obtain from the docker ps command from the beginning) to verify CPU usage. Same is true for RAM, for example:
core@ip-10-0-3-43 $ cd /sys/fs/cgroup/memory/system.slice/docker-0fb8849a255dcf3f5a80186da9e7866e5a6d48c05ff2bf7009601e5f3e7fb403.scope
core@ip-10-0-3-43 $ cat memory.max_usage_in_bytes
31711232
When we launch unicon-docker.json as a Marathon app the following is happening on agent ip-10-0-3-43:
core@ip-10-0-3-43 $ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
As expected, using the universal containerizer results in no Docker container running under the control of the Docker engine.
However, as usual, a Mesos sandbox /var/lib/mesos/slave/slaves/6522d1b9-0806-41c6-a0fb-b82f20c3c038-S1/frameworks/6522d1b9-0806-41c6-a0fb-b82f20c3c038-0000/executors/unicon-docker.01297b08-8e42-11e6-985c-e2bc47a71b50/runs/46d7cb05-ad44-4dc1-9f6b-2880bb287bd8 was created with a stdout containing worldview-unicon-docker.txt.
When we do systemd-cgtop on agent ip-10-0-3-43:
Control Group Tasks %CPU Memory Input/s Output/s
/ - 4.4 2.4G - -
/system.slice 392 2.8 2.4G - -
/system.slice/dcos-minuteman.service 90 2.1 138.0M - -
/user.slice 4 0.9 3.6M - -
/mesos - 0.7 72.2M - -
/system.slice/dcos-navstar.service 90 0.5 85.6M - -
/mesos/slave - 0.4 49.8M - -
/mesos/46d7cb05-ad44-4dc1-9f6b-2880bb287bd8 - 0.4 21.5M - -
/system.slice/dcos-spartan.service 126 0.1 98.6M - -
/system.slice/systemd-journald.service 1 0.1 131.4M - -
/system.slice/dcos-pkgpanda-api.service 2 0.0 28.7M - -
/system.slice/docker.service 15 0.0 248.4M - -
/system.slice/dcos-epmd.service 1 0.0 244.0K - -
/init.scope 1 - 10.0M - -
/init.scope/system.slice - - 3.3M - -
/system.slice/dbus.service 2 - 2.1M - -
/system.slice/dcos-3dt.service 8 - 10.3M - -
/system.slice/dcos-adminrouter-agent.service 2 - 3.1M - -
/system.slice/dcos-mesos-slave.service 34 - 6.9M - -
/system.slice/dcos-rexray.service 9 - 7.2M - -
/system.slice/polkit.service 5 - 10.0M - -
/system.slice/proc-sys-fs-binfmt_misc.mount - - 80.0K - -
/system.slice/system-getty.slice 1 - - - -
/system.slice/system-getty.slice/getty@tty1.service 1 - - - -
/system.slice/system-serial\x2dgetty.slice 1 - - - -
/system.slice/system-serial\x2dgetty.slice/serial-getty@ttyS0.service 1 - - - -
/system.slice/system-sshd.slice - - 2.8M - -
/system.slice/system-system\x2dcloudinit.slice - - 6.9M - -
/system.slice/systemd-logind.service 1 - 1.6M - -
/system.slice/systemd-networkd.service 1 - 1.9M - -
/system.slice/systemd-timesyncd.service 2 - 868.0K - -
/system.slice/systemd-udevd.service 1 - 17.2M - -
/user.slice/user-500.slice 4 - - - -
/user.slice/user-500.slice/session-3.scope 4 - - - -
Using the cgroup /mesos/46d7cb05-ad44-4dc1-9f6b-2880bb287bd8 (as was the case with in the frist example, named after the Mesos task ID) we can have a look at the memory usage:
core@ip-10-0-3-43 $ cd /sys/fs/cgroup/memory//mesos/46d7cb05-ad44-4dc1-9f6b-2880bb287bd8
core@ip-10-0-3-43 $ cat memory.max_usage_in_bytes
32567296