Github actions: How to capture a crash dump

gh_actions_crash.md

If an application is crashing during a github actions run then you can use this guide to capture the core as an artifact for debugging.

This guide was developed to debug Mac crashes specifically. It should mostly apply to Linux as well but getting dumps from Windows is unknown for now.

Mac / Linux

By default these systems will not be in a state where they are recording core dumps, and even when enabled you need to run with elevated permissions for a core to even be written. To capture dumps you need to do the following:

1. Enable core dumps with 'ulimit -c unlimited'. This command need to be in the same run block as the crashing program since it reset for each step.
2. Execute you program with 'sudo'. Core dumps will not be written without elevated permissions.
3. Enable access to core dumps by chmodding their locations. On Mac they are locaed in '/cores'. On Linux I think they go in '/var/crash' but I haven't tested it. They may end up somewhere else due to apport.
4. Upload dumps as artifacts. They will be huge. On mac they were 2GB.

    - name: step with crash
      run: |
        ulimit -c unlimited         # Enable core dumps to be captured (must be in same run block) 
        sudo your_command_to_run    # Excute with elemvated permissions (works with scripts)
        sudo chmod -R +rwx /cores/* # Enable access to core dumps (doesn't need to be in same run block)
    - uses: actions/upload-artifact@master # capture all crashes as build artifacts
      with:
        name: crashes
        path: /cores

If anything isn't working there's a few steps you can go through to double check everything is working:

Test captures are enabled:

    - name: test enabling cores
        ulimit -c               # should output 0 if disabled
        ulimit -c unlimited
        ulimit -c               # should output 'unlimited' now

Test Access to core dump directory:

    - name: test access to core dump directory
        sudo touch /cores/test
        ls /cores
        sudo rm /cores/test

@kudaba: Thanks a lot for your handy guide. 🤗

I have a core dump which is about 3.2GB, and the Github artifact upload seems to be very slow and often reaches the github actions timeout. Did you have problems with uploading your 2GB core dump, or was it pretty fast?

I don't remember it being noticeably slow but it was a long time ago and I only captured 3-5 dumps for a very small process :(

The timeout if something you can control per job, though, so bumping while tracking a crash seems like the way to go, unless there's a global timeout that I'm forgetting about. I Usually comment out all jobs in a separate branch to focus on the just the crashing part so I don't pay full CI cost while trying to fix it.

I'm trying this without success. The test to check the core dump directory doesn't return anything, even though the test file is later available in the artifacts. However, the core file is not. I'm quite confused.

I haven't tested this in years so there's a good chance something changed. If you do figure it out, please leave a comment so we can keep this up to date.

I could not get it to actually do the "core dumped". It just stays at segmention fault, not sure why. I'm now trying to get a backtrace using C++ ways of catching the signal and printing the backtrace instead.

The default for core dumps is piping them into apport, so you'll have to change the core_pattern file for creation the core dump under /cores.
Something like this will do the trick:

          sudo bash -c 'echo "/cores/core.%e.%t.%p" > /proc/sys/kernel/core_pattern'

When trying

          sudo bash -c 'echo "/cores/core.%e.%t.%p" > /proc/sys/kernel/core_pattern'

In the ubuntu-latest Actions runner, will return a permission denied