Skip to content

Instantly share code, notes, and snippets.

@skuenzli
Last active January 4, 2018 15:37
Show Gist options
  • Save skuenzli/3a52273febefb84efe470fbdad92fa97 to your computer and use it in GitHub Desktop.
Save skuenzli/3a52273febefb84efe470fbdad92fa97 to your computer and use it in GitHub Desktop.
Generate core files for crashed processes in containers

Overview

This is a simple demonstration of generating core files for crashing processes that happen to be containerized to a known location on CentOS 7.

Terminal 1 - Configure System and Run Process

Configure Linux kernel with a core pattern that puts files at a well-known location.

The location will be interpreted from the crashing process' perspective of file paths [1]

sudo sysctl kernel.core_pattern='/tmp/cores/core.%e.%p.%t'

create a directory to store core files that can be bind-mounted into containers at the known location

mkdir -p /tmp/cores/

run a container with the cores directory mounted:

sudo docker run -it --name crashy -v /tmp/cores:/tmp/cores centos:7.4.1708 bash

run a victim process in the container

sleep 1d

Terminal 2 - Crash the Victim

In a second terminal on the container host (not container)...

Crash the victim process and generate a core using the ABRT signal:

sudo kill -ABRT $(pgrep sleep)

list the contents of the cores directory ls -la /tmp/cores/

should produce something like:

[centos@ip-172-31-16-47 ~]$ ls -la /tmp/cores/
total 148
drwxr-xr-x. 2 root root     38 Jan  4 14:21 .
drwxrwxrwt. 9 root root    185 Jan  4 14:21 ..
-rw-------. 1 root root 380928 Jan  4 14:21 core.sleep.13.1515075705

Notes

There are at least a couple hidden reasons for why this works:

the container was run as the root user and...

  • inherited ulimits allowing an unlimited core file size
  • has permission to write to /tmp/cores/ on the container host filesystem

In a production setup, containers generally won't (shouldn't!) be run as the root user, so care needs to be taken that the proper ulimit and filesystem permissions are available for the user used by the containerized process.

References

[1] See 'Paths are interpreted...' in Linux man page for core files

[2] Documentation: Make clear instructions for getting a core file, when container crashes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment