1st November, 2012
Luckily we have got a beta access to Travis CI pro which allows you to build your private repos on github. I did some research for my work and we have decided to replace our Jenkins to Travis CI pro. I leave some notes here.
We have bunch of private repositories, mainly Ruby (rails and gems), on github that we want to continuously test. We currently maintain two CI servers: Buildbot for legacy project and Jenkins for recent projects. On this research, I simply ignored the legacy one and evaluated Travis CI pro as an alternative for Jenkins.
Our Jenkins is managed by Chef and quite scalable. We can add a slave (or build master) on EC2 with typing a few commands on terminal. So to maintain Jenkins is not big deal for us. I was skeptical to switch Travis CI before I gave Travis CI pro a try since Jenkins does a good job to us.
First impression: it is AMAZING.
I have been using Travis CI for open source projects. I knew that it is good but the experience on pro blew my mind again. It is dead simple and easy. What you have to do is two things:
- Add
.travis.ymlon your repo (very easy) - Enable build for the repository on admin screen on Travis CI pro
It is far easier than adding a new project on Jenkins. I love it.
It is not so bad. It is actually much better than I expected. Test is faster than running same thing on EC2 large plan. However it takes some time on bundle install. When we run tests both on our Jenkins and Travis, our Jenkins usually gives us a shout on our HipChat before Travis does. But it is slight difference.
Besides Travis CI pro is a still in beta and that's something they can improve later.
(Disclaimer: I didn't take any serious benchmark for performance evaluation. Take this information just as one case study)
We use HipChat notification. You can set a room (or rooms) to be reported for each repository. Again, it is very easy to set up.
I also love its email notification which sends only to people who broke the build. Team can also keep their eye on hipchat and Travis CI screen. The person who needs immediate action is only a guy who ruined. Your inbox is clean as long as you're not the guy :)
(Note that it uses email in commit log. Don't forget to set email address you want use with git config. Took while to notice this and wondered why it didn't use my work email address)
No configuration is required. Travis CI tracks branches automatically and tell you whether the pull request breaks build or not. It's nicely integrated with github.
I think that $129/month for two concurrent builds is very reasonable. It is cheaper than EC2 host with large plan. This makes sense because we would not run test 24 hours per day. Service like Travis CI can use cloud resource effectively.
Besides you can expect to reduce some maintenance cost for human resource too.
Yes, it is true that Jenkins allows you to do more than with Travis CI. However that ability has actually been bugging me... You can easily spend a few hours to find right settings/plug-in (also a lot of forks out there!) for your Jenkins.
When I research something, to know that I can not do what I want to do as quick as possible is also important. From that point of view, Travis does a good job.
Travis CI just supports what important to us for CI server.
The biggest concern for me is that your tests always have to run on their environment.
http://about.travis-ci.org/docs/user/ci-environment/
In fact, Travis CI could not build one of our repos because the version of phantom.js on Travis is not the latest. Their support team said that they plan to upgrade it next week and fortunatelly that repo is still in early alpha so that we could wait. But I am still worried the fact that you have to stick with the environment they provide. I am not sure how annoying it could be in practice in future. We will see.
(Update: https://twitter.com/svenfuchs/status/264050760709451777)
Travis CI pro is awesome. It doesn't give you any pains which you usually have when you set up CI server. It just gives you a pleasure which you have when you see green light. I recommend you to give it a try if your private repos are on github and your repos fit their server environment.
By Tatsuya Ono
We ran into an issue with the PhantomJS version too and found there was a pretty straight forward fix. Check it out here. http://collectiveidea.com/blog/archives/2012/10/17/install-your-own-phantomjs-on-travis-ci/