BOSH Turns 5

blog.md

It is no fun to write complex open source that no one can deploy on Day 1 nor can continue to live with it on Day 2 onwards.

Five years ago the incredible lifecycle orchestration platform called BOSH was open sourced by VMWare. It changed the game for owning and operating complex systems and is unparallelled even in 2017. Five years later - more feature rich and easier to use - it is the preferred packaging and deployment system for Cloud Foundry, Kubo (Kubernetes distribution), Pivotal Network, Concourse CI, Dingo PostgrSQL (PostgreSQL as a service), Logsearch (Logstash distribution), and others.

I was at VMWare on April 11 2012 when BOSH was first open sourced, and it makes me very happy to still be successfully running our own systems and our customers systems with BOSH five years later. In this article I wanted to share a reminder of what 2012 was like, why BOSH came into existence, and how much it has improved over the last five years.

Wind the clock back to April 2011, six years ago. The king of proprietary virtualization, VMWare, revealed a brand new platform for web developers to deploy applications called Cloud Foundry, and they open sourced it to boot. It was a complex system - bespoke Ruby and Java applications and OSS databases. To deploy this system you would use the state of the art at the time: Chef recipes.

Anyone who successfully deployed Cloud Foundry discovered that the "how to deploy" phase was only the gateway into a lifetime of "Day 2" problems:

• how do you upgrade the entire system to a new version?

• how do you test the upgrade of an entire system from one known version to another?

• how do you package the entire system into versioned software that can be systematically deployed?

• how do you resize host machines - replacing older/smaller servers with bigger/newer types?

• how do you resize persistent disk volumes?

• how do you rotate host machines with new base images containing zero day security fixes?

• how do you repave host machines every few hours or days to limit the processing time available to hackers?

• how do you automatically resurrect missing host machines, replacing them with new machines and reattaching persistent volumes to return data services back into operation with minimum downtime?

To solve these Day 2 problems, BOSH took ownership of a vast slice of the problem space - other solutions had/have only tackled smaller slices. Terraform can provision persistent disk volumes, but cannot mount/remount within host machines and cannot perform a resizing function. Docker can package software into a concrete version, but could not manage the lifecycle of host machines (recently infrakit has been proposed to help with this). Chef, Puppet, Ansible solve configuration management but not packaging; and package management tools do the reverse.

BOSH took complete responsibility for running a system - from Day 1 provisioning of initial resources, package management, configuration management. BOSH also too responsibility for Day 2 problems above (rolling repavement of host machines is exciting work in progress).

Instead of keeping BOSH to themselves, VMWare's Cloud Foundry team announced that BOSH was now open source on April 11 2012, five years ago.

Five years ago, I was looking for a holistic solution to all the problems above and so I grabbed BOSH with both hands. During the same five years, the devops industry has been the beneficiaries of incredible set of tools, solutions and new communities: Chef Habitat, Docker, Hashicorp, Kubernetes, Mesos, and more. We've had the rise and ... continued existence of Open Stack. Azure and Google Cloud Platform are common preferred cloud infrastructures for many large companies. Amazon AWS has grown in size and grown in its offerings. It is an incredible time to be a productive systems/platform operator. To be "doing devops".

Five years later, I observe that there is still no competing solution to Day 1 and Day 2 problems of running systems. Nearly all Cloud Foundry deployments rely on BOSH. Kubernetes distribution Kubo relies on BOSH. Stark & Wayne's own PostgreSQL-as-a-service Dingo PostgreSQL relies on BOSH. A consistent platform for dev/test of complex systems and for sharing those systems with confidence that people will be able to run them.

It is no fun to write complex open source that no one can deploy on Day 1 nor can continue to live with it on Day 2 onwards.

BOSH in 2017 is wonderfully more simple to operate than in 2012. And yet so many changes over the last five years have been on making it even better for running large complex production systems.

To recap my favourite improvements over the last five years:

1. The bosh CLI has been rewritten in Go and is distributed as a compiled binary. Its hugely faster, and its a lot easier to install with confidence. Previously bosh was distributed as a Rubygem, it meant that you first needed to install Ruby & Rubygems. Many system admins did not use Ruby. They would attempt to install Ruby like any other package; but apt-get install ruby never gets you the version of Ruby you want. Remember: friends don't let friends install programming languages from package managers.

2. Adding security throughout your complex systems is now easier to do than not to. There is a huge push in the BOSH community to do away with the idea of "skip certificate verification" and weak passwords. BOSH now makes it very easy to generate certificates and passwords and to share them throughout the system. In future it will make it easier to rotate these between deployments. Its like mathematical induction for security: I don't care about my initial passwords and secrets; and I don't care if you change them. So, by mathematical induction, at all times I don't care about my passwords. Keep on changing them so no one else tries to care about them either.

   Fun fact; for much of the last five years, the default SSH access to every BOSH VM on the planet was vcap / c1oudc0w. BOSH VMs no longer have known-in-advance SSH access.

3. Public clouds all have public AMIs. Five years ago, if you wanted to use BOSH with AWS you were given all the tools to create your own AMIs - which was personally a huge gift, since such tools didn't exist (Hashicorp Packer didn't exist yet for example). Today, the BOSH Core team maintains a CI pipeline that publishes public AMIs ("light stemcells" in BOSH language) for public clouds.

4. bosh.io/docs is a thing. OMG, when I was getting started there were no docs.

5. bosh.io/releases is a community repository of BOSH releases - nearly anything you'd want to deploy is available, with inline documentation to help getting started

6. bosh-deployment repository is an all-in-one starting point for deploying a BOSH director (BOSH is first and foremost a running server itself, with a bosh CLI to interact with it) to a public cloud, vSphere, Open Stack, or even your local machine via Virtual Box. Learn to use BOSH on Virtual Box and you'll have very much the same user experience running a massive system on a public cloud or vSphere.

7. BOSH 2.0 features such as Links, Cloud Config, Config Server, and more have made it so much easier to write a BOSH release and share a single deployment manifest. It is easier than every for an a multi-host OSS project to help their community actually deploy and run their project in production.

8. BOSH is no longer known only to Cloud Foundry community. It's awesome to hear stories of companies who've "BOSH-ified" their internal software and are excited by "life with BOSH".

The last five years of hardening, innovation and user experience niceties would never have been possible without huge R&D investments by Pivotal, IBM and many other companies within the Cloud Foundry Foundation.

As a consultancy firm, Stark & Wayne is also thankful to all our clients who've willingly and enthusiastically allowed us to open source tools and to upstream patches to core BOSH & Cloud Foundry. We live within a wonderful community of contributors and corporate users.

BOSH has over 1000 members of the #bosh Slack channel (https://slack.cloudfoundry.org). You'll find all the help you need from the community, the core team members, and our stoic leader: Dmitriy Kalinin. For years Dmitriy has been guiding BOSH into the great shape its now in. What a champion.

Our profession of building, deploying, nurturing and securing production systems is only getting more exciting and productive. BOSH is a power tool that you should definitely add to your team.