Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Haskell projects with lots of dependencies on Travis CI

If your Haskell project has lots of dependencies, you can find yourself reaching the timeout while compiling them - the free service has a timeout of 50 minutes. How do we address this?

First, a quick reminder of build statuses in Travis CI:

All builds finish with one of four possible statuses: succeeded, failed, errored, or (manually) cancelled. The difference between a failed and errored build is that a build is marked as errored if any of the build steps in the install phase fails, whereas a failed build is where one of the build steps in the build phase fails.

The general idea is to address the issue of builds timing out by allowing the next build to pick up where the previous one left off, by ensuring that the cache is preserved between the builds.

This solution is based on a few observations:

  1. Storing the build cache does count as part of the build as far as timeouts are concerned - if we want the build cache to be stored, we need to make sure the build finishes with a little time to spare for uploading the new cache.
  2. The build cache is stored on succeeded and failed builds, but not errored builds. (Note: this may change in the future, see: https://github.com/travis-ci/travis-ci/issues/4472)
  3. If a build is restarted, it starts with the same version of the cache as it did the first time. So if we want to pick up where we left off, we can't use the 'Restart build' button in Travis' web UI, we need to trigger a new build by pushing a new commit.
  4. Once the cache is set up, builds are relatively quick. It is fairly rare that so many dependencies must be rebuilt that we do hit the timeout - the only occasions I am aware of that might cause this are switching Stackage snapshots or GHC versions. So it's acceptable if it's a little bit of a faff to get the cache set up again, as long as it's easy.

So how does it work? The key is a handy tool called timeout, which is part of GNU coreutils. This tool can run a command but kill it if it fails to complete in a specified amount of time.

One thing to be careful of is that subsequent build steps are still run even if one build step fails. So I recommend writing a separate bash script and having your build phase in .travis.yml look like this:

script:
  - ./travis-script.sh

Then, inside travis.script.sh, have something like this:

#!/bin/bash

set -x

# Install GHC and build dependencies
timeout 40m stack --no-terminal --jobs=1 --install-ghc build --only-dependencies
ret=$?
case "$ret" in
  0)
    # continue
    ;;
  124)
    echo "Timed out while installing dependencies."
    echo "Try building again by pushing a new commit."
    exit 1
    ;;
  *)
    echo "Failed to install dependencies; stack exited with $ret"
    exit "$ret"
    ;;
esac

# Build your project
stack --no-terminal --jobs=1 build --pedantic

For Travis OSX builds, you can run brew update && brew install coreutils and then the timeout tool will be available as gtimeout. I'm not sure if the brew update is necessary but the Travis docs recommend it.

@hdgarrood

This comment has been minimized.

Copy link
Owner Author

commented Feb 8, 2017

Also, I need to try with different --jobs values. I think 2, 3, or 4 might be better options than 1. Leaving it unset tends to not be good, though, as then GHC's runtime will ask Travis how many CPUs it has to work out how many "capabilities" it should configure itself to use, and Travis tends to respond with a number much larger than what it will actually allow any single job to use.

@hdgarrood

This comment has been minimized.

Copy link
Owner Author

commented Feb 8, 2017

You might also want to increase the cache.timeout setting in .travis.yml, as Haskell build caches can get rather large and therefore take a while to upload. See: https://docs.travis-ci.com/user/caching/#Setting-the-timeout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.