gistfile1.md

Travis CI consists of two main parts right now:

• the web app which is a regular rails app
• workers which execute the tests

As we are moving to AMQP for communication between app and workers we want to split up the web app into

• a rather "dumb" asset-delivery and api service and
• a message hub that schedules jobs for the workers and consumes all the messages that come back from them (currently misleadingly called "consumer", as it consumes but also publishes)

We want to do that in order to (potentially, if necessary) be able to scale up the "consumer" horizontally and also simply separate concerns (currently the web app handles tons of things, deployments and tests are rather slow, etc) and make the web app simpler.

Domain logic in Travis CI includes things like the following:

• when a ping from Github comes in, create a bunch of objects (Request, Commit, Task::Configure) and put the Task::Configure (will be renamed to Job::Configure because it's a "job-to-be-processed-on-a-worker") on a queue
• when the configure job is done save the resulting configuration, some timestamps and create a bunch of Task::Test (jobs that actually perform tests on the workers) depending on the configuration
• while the tests run on the workers, results (i.e. log updates) are continuously streamed back to the app and stored
• when a test job is finished, store the result and a timestamp
• when all tests that belong to a build are finished, then finish the build, calculate the final result and build duration etc.

We are using a statemachine-like helper for wiring up all the state changes (https://github.com/svenfuchs/simple_states). This gem can be used with ActiveRecord but also with plain Ruby objects (e.g. https://github.com/travis-ci/travis-ci/blob/master/app/models/build.rb#L8-12, https://github.com/travis-ci/travis-ci/blob/master/app/models/task/test.rb#L5-8)

This state changing "process" logic will trigger various events/notifications that are emitted, like Websocket messages (continuously updating the JS client in the browser), E-Mail/IRC/Webhook notifications etc.

All of this logic could in theory be extracted from the web app easily and put to the new consumer app.

But the web app would still need to have all ActiveRecord models involved (like Build, Task::Configure, Task::Test, Request, Commit etc) because it needs to deliver them on the JSON Api and similar things.

Two main questions:

1. Both web app and (new) consumer app would need to share the database, its schema (migrations?) and the ActiveRecords (in some way). How to go about this?
2. The whole state changes logic could be separated from the DB storage logic and the AR model relations. Would that make sense?

One solution would be:

• Simply duplicate the ActiveRecord classes graph as well as migrations and the db schema across the web app and the consumer app.

Another solution could be:

• Extract the ActiveRecord classes to a separate gem "travis-records", which is used by both the web app and the consuper app.
• Wrap these record classes into a state-changes-handling and event-emitting class in the consumer app.

As we're touching quite some portions of the codebase I also want to look at and try out more modern testing practices in order to speed up and improve the quality of the test suites in general.

Wdyt? Comments?

Are there any resources we can look at for this? Any example apps?