Hosted and self-maintained continuous integration systems - CI recommendations by Rafał Pocztarski

ci-systems.md

Here are links to some reports, my own projects and my opinions regarding CI systems - mostly Travis, Circle and Jenkins.

Reports

Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects by Oregon State University and University of Illinois:

• http://cope.eecs.oregonstate.edu/CISurvey/

Comparisons

Jenkins vs. Travis CI vs. CircleCI on Stackshare:

• https://stackshare.io/stackups/circleci-vs-jenkins-vs-travis-ci

Best Hosted Continuous Integration Services for a Private Repository by Yegor Bugayenko:

• http://www.yegor256.com/2014/10/05/ten-hosted-continuous-integration-services.html

Continuous Integration. CircleCI vs Travis CI vs Jenkins by Roman Gaponov:

• https://hackernoon.com/continuous-integration-circleci-vs-travis-ci-vs-jenkins-41a1c2bd95f5

Continuous Integration. CircleCI vs Travis CI vs Jenkins by Alexander Shaporda:

Travis vs. Jenkins on Slant:

• https://www.slant.co/versus/627/2477/~travis_vs_jenkins

Circle vs. Travis on Slant:

• https://www.slant.co/versus/625/627/~circleci_vs_travis

Jenkins vs. Circle vs. Travis on Slant:

• https://www.slant.co/topics/1707/versus/~jenkins_vs_travis-ci_vs_circle-ci

My projects

Example configurations for Travis and Circle in my own projects.

The reports are available from the badges and links in the repos.

Travis - multiple runtimes

Using Travis with multiple runtimes:

• https://github.com/rsp/node-caught

Travis config:

language: node_js
node_js:
  - 7
  - 6
  - 5
  - 4
script: npm test

It's easy to use a test matrix in Travis - e.g. use 3 different runtimes, with 5 different configurations each.

Travis and Circle - simple

Using Travis and Circle for a simple Node project:

• https://github.com/rsp/ifm-lw-r1-js

Travis config:

language: node_js
node_js:
  - "8"

Circle config:

machine:
  node:
    version: "8"

Travis and Circle - custom dependencies

Using Travis and Circle for a Scheme project that needs installing a custom compiler:

• https://github.com/rsp/ifm-lw-r1-scheme

Travis config:

language: c
addons:
  apt:
    packages:
      - guile-2.0
script:
  - guile tests.scm

Circle config:

dependencies:
  pre:
    - sudo apt-get install guile-2.0
test:
  override:
    - guile tests.scm

Travis is more declarative (I list what package I need to have).

Circle is more imperative (I list commands needed to install the packages that I need).

My opinion

All of the popular CI systems are free for open source projects so we can choose anything we like but 90% of open source projects use Travis anyway. (According to the study by Oregon State University and University of Illinois and it's compatible with my own experience)

For hosted solutions, Travis in the CI systems is like GitHub in source control. We have Circle like we have Bitbucket and in principle all the features are there but practically all of open source projects prefer GitHub to Bitbucket and Travis to Circle.

It's like prefering iPhone to Android - it's hard to point to a feature that the competitor doesn't have, other than the overall look and feel, performance and ease of integration with other tools. Third party services tend to support GitHub and Travis faster than Bitbucket and Circle, like iPhone apps tend to be published sooner than the same apps for Android, even though there is no any hard rule for that.

From my experience, Travis is by far the best hosted CI system that I've used. It's like iPhone in a sense that it has all the features and integrations I need and everything is where I expect it to be.

Instead of using GitHub or Bitbucket we can also maintain our own Git servers that will do the job but with significant administration overhead and without the ease of integration with other popular services but with greater flexibility and possibility that, in principle, we can build a better system than GitHub, given enough time and resources.

The same is true for CI systems - instead of Travis or Circle we can maintain our own CI servers with Jenkins with the same pros and cons as hosting our own Git servers or mail servers. In principle we can do a better job than Travis but in practice it requires time and resources.

My recommendation

We should keep using Jenkins for all systems that currently use Jenkins.

We should use Travis for some new projects where it makes sense, and for some existing projects that currently use Circle.

Currently we're using the free plan of Circle with 1 concurrent build.

Travis and Circle probably make sense only for frontend and backend projects that need good integrations with services like Heroku or Netlify. Other projects that use custom build process should use Jenkins.

I would use Travis for the most critical projects and keep using Circle for less critical projects to make sure that we don't have problems with the limit of 1 concurrent build.

Travis will not do anything that cannot be done with Circle but it is more robust and convenient to use, and additionally it's important to have experience with all of the modern DevOps tools, services and workflows.

The only disadvantage of adding Travis to our tool belt is the cost of $69 per month.

Prices

• Travis: https://travis-ci.com/plans ($69 per 1st concurrent build, every next one is cheaper)
• Circle: https://circleci.com/pricing ($50 per 1 concurrent build, the 1st one is free)

TL;DR

Travis == iPhone / GitHub / Gmail

Circle == Android / Bitbucket / Yahoo Mail

Jenkins == ZeroPhone / custom git server / custom mail server

Circle is cheap Travis.

It's important to have experience with modern tooling.