Skip to content

Instantly share code, notes, and snippets.

@adamralph
Last active September 9, 2019 21:58
Show Gist options
  • Save adamralph/fdf68c8bfe7e1aa8d56f89df05ae15ce to your computer and use it in GitHub Desktop.
Save adamralph/fdf68c8bfe7e1aa8d56f89df05ae15ce to your computer and use it in GitHub Desktop.
My current assessment of SaaS CI systems

This assessment is performed with respect to the requirements of the .NET community open source projects which I maintain. It is restricted to the free tier provided by each system.

Only the drawbacks are listed, which makes the assessment sound a bit negative, but it's relative to a baseline functional expectation:

  • Linux and Windows
  • Fast provisioning
  • Concurrent builds
  • Common build properties surfaced as env vars (e.g build number)
  • Readable, copyable, and line-linkable log output, supporting ANSI colour codes and code page 437

Plus generally expected things like "a good UI".

  • Appveyor
    • very slow provisioning, especially of Linux agents
    • does not run concurrent builds
    • does not support code page 437 on Windows without workarounds
  • Azure DevOps
    • console output is utterly broken, in too many ways to list
    • navigation (and UI in general) is painful
  • Circle CI
    • doesn't support Windows
    • console output:
      • is not retained - it's only shown if the page was opened while the build was running
      • is not line-linkable
      • often has an extra blank line between every line
  • Cirrus CI
    • doesn't surface a build number
    • console output does not show any whitespace in lines with ANSI colour codes
  • GitHub CI
    • doesn't surface a build number
    • console output is not copyable - try selecting a section of text larger than the visible buffer!
  • GitLab CI
    • doesn't support Windows
    • console output is not line-linkable
    • no further investigation
  • Travis CI
    • only has partial Windows support
    • slow provisioning
    • only makes env vars available after parsing the YAML file, which is a complete PITA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment