Dependency Part One

gistfile1.md

#Berliner and Core - The New Road...#

The recent changes to the Responsive News Codebase have raised some questions and concerns that we are going to try and address below.

##Why?##

1. The most important reason in terms of immediate benefit that can be felt by everyone is the improvement in Build Speed. This can be split up into three results which we are already benefitting from:
   1. Our releases / deployments now include less RPMs, by default, meaning that our build process is shorter.
   2. This reduction in build "weight" means that the feedback cycle for engineers who are integrating new code is shorter / quicker.
   3. And so in turn we can integrate more things more quickly, as long as they continue to be lighter touches and smaller, more atomic changes.
2. Architechtural Clarity. While there is no doubt that the Responsive News Codebase has always moved from a position of thoughtful technical design, it has become the consensus amongst the News Technical Leads that the future is going to necessitate an architectural (r)evolution, and as such one of the cornerstone concepts in relation to this is the idea that separating our various components into applications and / or libraries in their own right is the first step along the road to a system that has far greater clarity at its heart. Separating these two initial components out and the others that we intend to break out after them has either achieved the following things already or will empower us to make them a reality in the near future:
   1. Reducing the scope for unexpected behaviour.
   2. Allowing us to control both the "direction" of dependencies (or rather the shape of the dependency tree), and to ensure that all dependency-based relationships be formed based soley on giving applications / components the code that they need to work (none that they do not need), and that as far as possible the dependent code should be completely utilised by the application relying on it.
   3. Taking control of the dependency graph of our system in this pro-active manner is going to allow us to take the next evolutionary step in our codebase wherein we will have components that carry out their specific role or roles by providing the smallest set of functionality possible in order to meet the global requirement of their role(s).
   4. Imagine, if you will, a situation wherein at some point in the future we will have an application component which has the sole concern of assembling and rendering Story Pages. This application would need to satisfy its dependencies by using Core and Berliner as libraries, but would have no need of any other code form the wider Codebase. This application component could be deployed anywhere; Forge, AWS / Cosmos, A.N. Other Cloud Provider, and because its dependencies are satisfied by stable, packaged libraries it can be deployed at need, alone, without needing to deploy any other application or component form the Responsive News Codebase. That is the endgame which we hope to facilitate with this change.

##How?##

1. Composer...
   1. OK, so what is Composer? Composer is a Dependency Management Tool for PHP. It is analogous to tools that exist for many programming languages, such as Bundler for Ruby or SBT for Scala or CPAN for Perl. It works by using an executable that does two things. The first thing is that it reads a config file (composer.json) and creates a lock file (composer.lock) that is a manifest of sorts for dependencies, based not only on the packages referenced in the composer.json file, but also the secondary and tertiary (and deeper, potentially) dependencies for them. The second thing is that using the lock file, Composer downloads and installs all of the relevant dependencies to the version required. This means that using Composer removes the risk of using dependencies, to the extent that it becomes responsible for ensuring that they are satisfied, and it makes it possible for us to control the shape of how our components and application relate to one another.
   2. At the moment we are operating a simplified dependency graph that makes Core and Berliner dependencies of Tabloid, and Tabloid a dependency of all other applications / components. This means that work on the Responsive News Codebase that does not touch Core or Berliner will be able to carry on as normal with only one change; the need to run "rake:composer:install:tabloid" in order to satisfy the dependencies required to run the Codebase on a Sandbox.
   3. On that basis, the promise that we are now making to the Engineering Team at large is that Core and Berliner are now stable libraries that fulfil the following requirements:
      1. They will never include breaking changes.
      2. They will maintain backwards compatibility.
   4. Should those promises become untenable for any reason, that adjudication will come from the Tech Leads group, and any break with either promise will be subject to complete visibility across the whole team, technical and non-technical, and will be made with enough notice and in line with priorities and schedules to ensure that the very minimum of disruption is caused. In the vast majority of cases if you want to make a change to Core or Berliner that would break these two promises there is a good chance that you are "doing it wrong".
   5. Final Note - If you are working on a change that has been approved on Berliner or Core, in concert with another change in the codebase and once that first change has been accepted, you will need to run "rake:composer:update:tabloid" in order to have your most recent changes to develop and test against in your sandbox. The Tech Leads group and the current Release Engineer will orchestrate the deployment of library changes such that no one else need care - please see following sections for more detail.

##Some thoughts on separate Git repositories##

1. From a simplistic stand point, the selection of Composer to manage our dependencies means that we need those "chunks" of code that will be managed in this way to be in separate repositories, so that Composer can treat them as discrete "chunks" of code and access them based on commit hashes and / or tags.
2. In the fullness of time we would like to get to a position where feature addition or enhancement, as well as maintenance may be achieved on a component by component basis and having the components that make up our platform live in their own repositories not only means that the version and change histories of each becomes specific to its own concerns, but also empowers further tooling that will undoubtedly rely on automation keyed from verison control, whether that continues to be Git or evolves to another paradigm over time. In other words the separation of our codebase into smaller codebases is a necessary step along the road to Continuous Delivery, and while that is a destination some way off for now, we are now in a place where we need to begin the ground work for that shift.

Very thorough Oli, nice :)