This is a series of notes on the Essay https://12factor.net/codebase
A twelve-factor app is always tracked in a version control system, such as Git, Mercurial, or Subversion. A copy of the revision tracking database is known as a code repository, often shortened to code repo or just repo.
A codebase is any single repo (in a centralized revision control system like Subversion), or any set of repos who share a root commit (in a decentralized revision control system like Git).
12 factor app specifically argues against a monorepo. Their argument is that the shared code should be refactored out into dependencies, as in Factor II. This is a valid argument, but the overhead of separating dependencies and testing them individually is not worth the cost for most small teams. The pattern of building a series of components of a distributed system from a common codebase - Of having one monorepo that produces multiple artifacts - is a good one.
The twelve factor app manifesto even contradicts itself here with it's advice: Factor XII makes clear that Administrative tasks are to be included in the repo, but by its own definition here those would qualify as components of a distributed system.
One of two important tasks here is to have clear definitions of which artifacts are produced and who owns them. Each team should be responsible for its own builds, and breaking others builds should be impossible - the CI system should reject changes that don't build, and should include code ownership tools to prevent changes to other teams "owned" code, including their artifacts definitions, without their approval.
The second consideration is to make clear the ownership of shared libraries. The codebase should operate on a strict "You break it, you fix it" paradigm - it's the responsibility of the one making changes to make sure they work and get approval from stakeholders on those changes. Your third development team should be a cross functional Infrastructure team, with the express remit of making the common code operate better.
The largest companies operate monorepos. A few dependencies in those are considered "Stable" and operate outside the monorepo, but most components are in tree. That doesn't mean they're constantly used - The large companies have pinned versions of core components, that are treated like vendored dependencies. Development on those core components is done carefully on branches, or preview versions available as alternately named packages, but the update is done by updating all of their dependants as a singular update.
In short, this is a tooling problem, either way, but there are significant attractions to a monorepo model
A twelve-factor app never relies on implicit existence of system-wide packages. It declares all dependencies, completely and exactly, via a dependency declaration manifest. Furthermore, it uses a dependency isolation tool during execution to ensure that no implicit dependencies “leak in” from the surrounding system. The full and explicit dependency specification is applied uniformly to both production and development.
This is more-or-less standard for modern packaging and deployment paradigms, but bears repeating. One of your most important dependencies is development environment. Build development environments in reproducible fashion - Test your "New team member setup" repeatedly. Don't let it rot. Likewise, spend time - Allocate a monthly or quarterly maintenance task to updating to the latest dependencies. Security and functionality bugs appear in every tech stack. Update frequently to the latest minor versions of your stack to take advantage of buggix/patch versions for as long as possible.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
There's three common methods of passing configuration into applications. Flags, Environment Variables, and Config files. They have security properties that are important on standard Unix systems. Flags passed on the command line are visible to all users on that system, which means they are a security risk. Secret keys or sensitive values should never be passed this way. Environment variables are visible only to other processes running as the same user - But they are easy to leak as they are inherited by subprocesses, and many common applications print some or all of their environment on startup. Files, with a mode of 400, should be the preferred way to pass secrets to your application. Flags defining where to find those files are fine.
A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.
Version control your configuration. The Twelve Factor app encourages you to avoid doing so because credentials are stored in your configuration. This is an anti-pattern. You should store configuration as configuration, and secrets as secrets. Secrets should live as files on your filesystem only populated at runtime, but all of the other things that need to be configured - Application root URL, feature flags, database locations, these should be stored in your version control system and treated as such.
Configuration completely as environment variables is a good pattern. It becomes unwieldy as the number of configuration options grows into the tens; It becomes untenable with configuration in the hundreds. You may well have a dozen backing services (see Factor IV), each with 3-4 configuration parameters.
The answer to this is to use a well structured configuration object. Treat this as any API input - Add new fields early before they are meaningful, ignore old fields (instead of making them errors) for several revisions. This configuration object should be loaded as a file from disk. The location of this file should itself be treated as a configuration file - it should be as simple to change from one set of configuration to the next by swapping out the flag. You should be able to run multiple copies of an application on the same machine by keeping multiple config files around (with appropriate changes to their configuration and port bindings, see Factor VII)
An example of the transition is Kubernetes. Kubernetes had a proliferation of flags, and switched to a Config File format. Kubernetes had a well maintained set of flags for configuration, ordered in a dot-separated hierarchy. Flags allowed sane defaults to be set effectively, but accumulated over time, even with aggressive pruning of deprecated feature flags. Eventually, they replaced the set of flags with a configuration struct constructed in yaml or json.
There's two common mechanisms for storing database credentials: To store a database URL, a username, and a password as three seperate fields, or to store them as one long string, e.g. postgresql://user:secret@localhost/otherdb?connect_timeout=10&application_name=myapp
The advantage of the former is that it's very easy to compare and differentiate the connection properties between instances of your application. The advantage of the latter is that there's one configuration and it's easy to manage. Pick one, stick with it, and document your reasons for connection parameters carefully.
The missing portion of this factor is that backing services should be properly organized: take care and make clear that you decide on the purpose of backing services and separate them by purpose. A backing service is not "Redis" or "Postgres", it is "Auth cookies" stored in Redis, or "User Accounts" stored in Postgres; you may have multiple backing services served by the same type of database (i.e. multiple Redis clusters or PostgreSQL instances).
It is possible (but uncommon! Understand your scale) to run into situations where you need more resources in your backing store than can be accommodated by a single machine. One solution is to split databases by table - To corral some tables into one database replication group/cluster, and others into another cluster. Draw this on your design diagram. Now you have a new problem - A proliferation of backing stores, each large on Production but still very tiny in development and staging: Make that clear, too. A backing store should not be "A RDS Instance", but rather "A Database on a Postgres Server" - And thus you can develop a schema for running many backing stores on the same installation of your database software, by creating developer1-objecttype1, developer1-objecttype2, developer2-objecttype1... databases.
The build stage is a transform which converts a code repo into an executable bundle known as a build. Using a version of the code at a commit specified by the deployment process, the build stage fetches vendors dependencies and compiles binaries and assets.
This section is poorly named. The interface between a build and a release is an artifact. Artifacts include both executables as well as static assets. Make clear which artifacts you build and how you release them. Your CI system should both build your artifacts are release them to staging, but separate those stages. The only thing passed between them is the name of the artifacts.
Better than immutable build numbers, tie your releases to version control hashes. Version control hashes are easy to forget and hard to order - Release numbers are easy to confuse and to misremember which changes are included. Hashes force your engineers to check your version control every time - And that's a good thing.
The only addition to make here is to reiterate the scope. This is for applications deployed to the wider web as a Software-as-a-service model. Your android app is stateful, and will have a stateful local backing store (usually a sqlite database).
Don't be picky about ports; any socket will do. A unix domain socket is a useful abstraction. It is unlikely to be used, and the most common usage for binding a domain socket instead of a port is the explicitly disfavored proxy webserver. Nevertheless, you will find circumstances where binding to a domain socket makes sense, and so if your framework supports it, accepting socket bindings as ":8080", "localhost:8080", or "unix://var/run/application/application.sock" can be handy.
Kubernetes and the kubelet are the "distributed process manager on a cloud platform" that is being referred to. Hashicorp Nomad fits the bill, and Mesos was an alternative but is effectively out of development.
Crash only design is a must, but so is lame-ducking. Your application should do it's best to complete requests in flight at the instant it receives a SIGTERM, and to gracefully inform clients that send requests afterwards to retry them against another instance. Healthchecks should immediately start failing. If easy, all requests after the SIGTERM should be responded to with a 503, and your frontend proxy should automatically retry them before returning that code to the client. If not, incoming requests should be responded to normally until the socket is closed. Graceful shutdown should happen as soon as all requests are handled.
There's an addendum to this that's too important to ignore: Ship the same Artifacts to Staging and Production. Don't build a "Prod artifact" and a "Staging artifact" - Build one artifact. A debug build is fine for staging, and you should have the option to build artifacts with additional symbols or tooling, but it should be one you're willing to ship to prod (if not replace all of prod with if there are performance constraints).
Any differences in Prod and Staging should be in how the binary is configured.
This applies equally to your consumer apps (Android, Electron, etc) - If you must build separate releases with different assets compiled in, build them from the same source at the same time. The prod build that ships should be the exact equivalent to the staging build that was validated.
There's two kinds of logs: Debugging logs, and Action Logs. Both are important, both can use the same framework, but understand the difference between the two.
Use a structured logging framework for both Debugging and Action logs.
Debugging logs represent internal state for informational purposes. They often include somewhat sensitive data, like session ids or internal representations of user state. They should be available for a limited time and then discarded. Aggregating
Action logs/Event Logs represent actions taken by the system or users of the system. An example of an action log is a login event; Say your application has regulatory requirements that it track the time, username, and IP Address of each login (This specific example is common but somewhat contrived: Regulations should equally prevent this information from being stored for certain classes of applications)
Practically, there's not good tooling out there for dealing with Action logs - Kubernetes provides no guarantees of any persistence to attempts to persist action logs. This means that that type of log will always be best effort. If you need precise, clear action logs - Log those events first into a backing store, and then serve them.
Admin Tasks should be built as artifacts, either as separate executables or as options when you run your application's executable. A fat JAR with additional main classes is a good option for an admin process, as would be a Python package with multiple main modules. A Docker image with both your application binary and admin processes included is reasonable as well, but could often just as easily be shipped as two images for saving in deployment time in the common case.
Your application should be observable as an application. It should include an OpenMetrics endpoint that can be scraped to provide key metrics - Such as counts of http response codes and histograms of their latency. You should also include key business metrics, such as "counts of shoppers" and "histograms of totals of successful checkouts".
In conclusion: The twelve factor app framework represents a large amount of best practices for the small-to-large businesses. There's a number of ways it overstates it's case and misses the mark, but it still provides a critical reference for anyone looking to build "Web-scale" applications, or even simple LOB applications with high user confidence and availability.