index.md

My Elixir Deployment Wishlist

Foreward

Based on my recent experience of deployment, I've become rather frustrated with the deployment tooling in Elixir. This document is the result of me thinking to myself, "I wish we had x...". This document isn't meant to dishearten anyone who has built tooling for elixir - thank you so much for what you've done. This is meant more as what I personally see as something that would help a lot of Erlang/Elixir newbies like myself to be able to get deploying quickly and efficiently.

1. Release files should be templates

It should be possible to add in custom configuration to the bootstrap scripts. This would allow plugins to be able to add extra steps to the startup / shutdown / upgrade procedure. One way to implement this would be to make all scripts which handle bootstrapping or controlling the machine .eex templates. This would allow other parts of the release system to inject new functionality where needed.

2. vm.args is generated at runtime

vm.args contains information about how the VM should run your elixir app. If you don't know what the server is like beforehand, it can be hard to produce a vm.args that will fit your needs. Creating the vm.args as part of the bootstrapping process would allow for more efficient use of BEAM, as well as configuration of things like the name and sname, which are often dependent on the hostname of the machine.

It's been established that this is actually already possible. Better documentation around this would be great.

3. Automatic init script generation

Init scripts will remain largely the same across projects. It makes sense to provide defaults for each init type, in the same way foreman works.

We agree that this is not the responsibility of exrm directly, but rather that of a plugin.

4. Release configuration should be held in an exs file

The release config file should be easy to configure, and match closely the patterns that people are used to following in config.exs.

5. Every step is overriddable and extendable

I think the phases of the release process should be simple to add to & reason about. I was imagining a config that has something like this in it:

def deploy_pipeline(foo) do
  foo
  |> PhoenixOverlay.compile_static_assets
  |> Exrm.build
  |> CustomOverlay.do_thing_a
  |> CustomOverlay.do_thing_b
  |> ExrmDebPlugin.package
  |> AptPlugin.release
end

Where PhoenixOverlay, ExrmDebPlugin and AptPlugin are External plugins, and CustomOverlay is some internal config specific to that app.

The release configuration file can override steps with custom steps, and also add new steps. In addition, plugin steps can be added to the pipeline.

5.1 Output Plugins

External Output plugins would be able to take a successful build, and do something with it (exrm-rpm is already capable of this). This could be:

• SCP the result to a server and upgrade
• Build a debian Package - DONE! exrm_deb now exists.
• Send a notification on Slack

5.1.1 Example A: Debian/RPM/Pacman Package plugin

Advantages over not being a plugin:

• Can use information from mix.exs automatically
• Can upgrades can be handled automatically through apt hooks
• Ability to create metapackages for umbrella apps

I've began work on a deb plugin, which you can check out here: https://github.com/johnhamelink/exrm-deb

5.1.2 Example B: SCP Deploy plugin

Advantages:

• Is able to detect when an update is happening automatically
• Simple to setup when deploying to a non-cloud VPS or bare metal

5.1.3 Example C: Slack Notification

Advantages:

• Can provide information about
• which apps changed (if in an umbrella)
• what environment they were built to deploy to

@johnhamelink Having to build a release on exactly the same architecture as deployment target has caused me most pain.

In reality it means pulling the source and building the release on the production box.

Perhaps this simply needs better documentation as I've struggled with building a release using both an EC2 instance and Virtual Box/Vagrant locally, both using same version of Erlang, Elixir and Ubuntu distro. Neither release would run on the target production box.

@johnhamelink I'm going to be replying piecemeal, as I think over some of these bits. First up, modular bootscripts:

I'd be happy to explore this! Would you be able to explain how you envisioned this would work? If you could propose a plan, I'd gladly attempt to submit a PR 😄

Sure, here's my general outline:

• Refactor the script into a lean core (basically arg-parsing, env setup, global helper functions)
• Extract command-handlers into sub-scripts which are imported in the core, to keep command-specific junk in their own files
• Introduce a plugin structure:
  • A plugin is a directory composed of shell scripts which are named after commands pre/post, i.e. pre_upgrade
  • When the core script runs, it determines what plugins are available by iterating a plugins directory relative to the boot script
  • When a command is executed, the pre- plugins are run, then the core command, then the post- plugins, by iterating over each of the plugins, determining whether a relevant command handler is available for that plugin and stage (pre/post)

This gives us the greatest amount of flexibility for extension, and easier maintenance due to better code organization, and division into smaller more maintainable units.

An example of the plugin bit I've just sketched up is something like the following (bear in mind this is super simplified to show the behaviour, implementation will rely on this basic mechanism, but will obviously look different):

# main.sh
#!/bin/sh

__modules_path="$(dirname "$0")/plugins"
SOME_VAR="foobar"

function __call_plugins() {
    __handler_to_call="$1"
    for plugin in $(ls "$__modules_path"); do
        if [ -d "$__modules_path/$plugin" ]; then
            if [ -f "$__modules_path/$plugin/$__handler_to_call" ]; then
                . "$__modules_path/$plugin/$__handler_to_call"
            fi
        fi
    done
}

function upgrade() {
    __call_plugins "pre_upgrade"
    echo "upgrading"
    __call_plugins "post_upgrade"
}

upgrade

# plugins/test/pre_upgrade
#!/bin/sh
echo "test: pre_upgrade: SOME_VAR=$SOME_VAR"

# plugins/test/post_upgrade
#!/bin/sh
echo "test: post_upgrade"

This all results in the following output:

$ ./main.sh
test: pre_upgrade: SOME_VAR=foobar
upgrading
test: post_upgrade

Thoughts?

@slashdotdash

Having to build a release on exactly the same architecture as deployment target has caused me most pain.
In reality it means pulling the source and building the release on the production box.

Cross compilation for another architecture is documented, and doesn't require you to build on the production box. You just pull the Erlang lib directory from the prod machine (or from a machine running the same OS/architecture) into your build machine, and provide a rel/relx.config with {include_erts, "path/to/erlang/lib"}.. Where you can run into trouble with this of course is with C extensions, where you need to provide the necessary cross-compilation toolchain for that extension. Some other alternatives:

• Run a build machine of the same architecture as your prod machine. Build your releases there.
• Run a docker VM of the same architecture as your prod machine, build the release, then export to wherever for deployment, or just deploy from the VM

Perhaps this simply needs better documentation as I've struggled with building a release using both an EC2 instance and Virtual Box/Vagrant locally, both using same version of Erlang, Elixir and Ubuntu distro. Neither release would run on the target production box.

Is this a reported issue I can look at? If the release won't run, even though the OS architecture is the same, then something else is likely the issue. At work, I run releases via ECS (I'm not doing hot upgrades), and haven't had any issues so far, though in that scenario I am running on my own image created via Docker.

@bitwalker Thank you for taking the time to reply.

I think you've actually identified the crucial step that I've been missing (as below, copying the Erlang lib from the target box to the build box).

You just pull the Erlang lib directory from the prod machine

Will have another go at creating a release with exrm on a separate build box with this in mind. Hopefully it will solve my pain point.

Here's my general outline:

• Refactor the script into a lean core (basically arg-parsing, env setup, global helper functions)
• Extract command-handlers into sub-scripts which are imported in the core, to keep command-specific junk in their own files
• Introduce a plugin structure:
  • A plugin is a directory composed of shell scripts which are named after commands pre/post, i.e. pre_upgrade
  • When the core script runs, it determines what plugins are available by iterating a plugins directory relative to the boot script
  • When a command is executed, the pre- plugins are run, then the core command, then the post- plugins, by iterating over each of the plugins, determining whether a relevant command handler is available for that plugin and stage (pre/post)

Ok, I like that strategy. I'll probably have a crack at this starting this on Friday. As far as things like vm.args runtime configuration, how would that work in this case? Could we run a .exs file that pulls in data about the system and produces a vm.args from a template?

I've also added @bitwalker to a private gitlab project I've created which is effectively a skeleton demonstrating how our system is currently built and deployed. I hope it further helps to explain where I am with exrm and how it fits into my current usecase. If anyone else wants access, let me know and I'll add them (I'd rather not make it public right now though).

@slashdotdash, @andrewvy has mentioned this as being desirable too, so good to know that it's possible. I wonder if we could precompile and keep on a repository somewhere so that a plugin could be created to re-run the mix release process across a matrix of architectures?

@johnhamelink I will follow up on your other notes this weekend, perhaps tomorrow if I have a chance, work is keeping me busy during the week though for the most part.

As for vm.args runtime configuration, I'm open to suggestions, but some things to be aware of:

• Once a release is running, vm.args changes won't take effect until the VM is restarted, so changes to this file won't be applied during hot upgrades/downgrades. My assumption is that this is fine, and templating out vm.args only needs to happen during start.
• I'd like to make this optional, and keep the current behaviour as the default

I saw your invite, and will give the project a more in-depth look when I sit down to go over your notes this weekend.

Regarding a means of making cross-compiling easier, if there was such a repository available, we could add a flag to the mix release task which fetches the necessary architecture, and automatically uses it instead of the system Erlang. Scripting releases for multiple platforms at once would be pretty easy from there. I think we'd want more of a file server type repository like hex.pm rather than using git for that kind of thing though.

Hey @bitwalker,

Just following up on your previous message - I know you're probably very busy, but did you get a chance to think about my notes above over the w/e?

RE cross-compilation, the wheels are in motion with regards to that. @andrewvy has began work on that, and my company has sponsored an S3 bucket for at least a few months to host the binaries in.

My question: would a repository of compiled ERTS for all architectures be helpful? I would imagine an accompanying Exrm plugin for pulling from the repository would be helpful. And also, what should be hosted? I'm not too familiar on architectures, should we compile ERTS for each distro?

Like this:

Or by architecture? x86, x86_64, armv5... etc.

---

We're tracking this through this github issue: https://github.com/erts-io/erts_web/issues/3

@johnhamelink That's great news! Thanks to you, @andrewvy and your company for taking that on! I'm going to take some time here to address your previous notes as well, following up shortly.

@andrewvy I'll reply in that issue thread :)

@johnhamelink Ok, going back to your first reply:

Hmm, just to clarify - which config file are you referring to in this case? I didn't make it clear I was referring to the next section.

I was referring to the relx.config file used today for configuring the behaviour of the release. Moving to a exrm_config.exs file would mean I'd include both the things that used to go in relx.config, as well as adding any new exrm-specific configuration. This config file would then be placed under the rel folder (I had quite a long discussion on this point in IRC awhile back, and it was made quite clear that people didn't want release configuration under config).

Where PhoenixOverlay, ExrmDebPlugin and AptPlugin are External plugins, and CustomOverlay is some internal config specific to that app. Does that provide better context?

The way I would actually lay the configuration out (in order to support some other features), would probably be something more like this:

# Would include the helper functions
use ReleaseManager.Config

# A single app release
release :foo,
  version: get_in_app(:foo, :version), # Basically calls get_in(Mix.Project.config, [:version]) for the app :foo
  include_erts: "path/to/erts",
  .. other relx.config legacy settings ..
  # These allow users to specify both raw functions or modules which adhere to the plugin behaviour,
  # they are then executed in the order defined, and could even be defined in this config file,
  # each function would receive the release Config struct as it's only parameter, and if returned modified,
  # will be passed modified to the next step
  before_release: [&PhoenixOverlay.compile_static_assets/1, SomePlugin],
  before_package: [&CustomOverlay.do_thing_a/1, &CustomOverlay.do_thing_b/2],
  after_package: [&ExrmDebPlugin.package/1, &AptPlugin.release/1]

# An example of an umbrella app packaged as a single release
release :foobar,
  version: "1.0.0"
  apps: [:foo, :bar]
  ...

That's interesting, I wasn't aware that this is the case. Why is that? Would it make more sense to build a dedicated metadata plugin that other plugins would use to query with, making it easier for plugins to be maintained as exrm (and BEAM, I guess) changes?

The primary lack of metadata I'm talking about is due to Mix not being available, even if you include it as an application in mix.exs, it still is limited in functionality because there is no mix.exs file with all your project's metadata. There is of course still the ability to get the release name, version, and static config (sys.config) for the app, so that's still a considerable amount, but not nearly the wealth of information that is represented by mix.exs (in my mind anyway). Ultimately we could probably just store plugin metadata in sys.config though, and use that for the source of truth. It wouldn't be available in to the running release (only configuration for loaded applications is accessible), but it could be queried from an escript with a simple :file.consult + get_in.

I really like way Ember handles deprecated code:

Release #1: Add the new functionality as a feature toggle
Release #2: Add a deprecation to the old functionality
Release #3: Switch the polarity of the feature toggle
Release #4: Remove old code

Definitely a fan of this approach, with some caveats. I think I would take a slightly different approach with the boot script though:

• Rewrite to use the modular structure, preserving old behaviour where possible
• Where not possible, raise an error if we can determine the old behaviour is relied on, with a message on how to migrate
• Where neither of the above are possible, document it as a breaking change, with clear instructions on migrating to the new version of exrm. Similar to how I did Timex 1.x -> 2.0.
• Add feature flags for the older, less-desirable behaviours we want to replace, automatically enabled, with deprecation warnings
• On the next release, disable all of the feature flags for older, less-desirable behaviour, but leave them there for those who still need them
• On the next release, remove the old code, but leave the feature flags in, raising errors if they are used
• On the next release, remove the feature flags entirely

So there is the potential that there would be a few breaking changes early in the process (though I can't think of anything that fits the description), but for the most part, everything would remain the same, with a very paced deprecation cycle. I dislike the idea of putting feature flags for new behaviour, and would rather do as I described, with old behaviour preserved via enabled-by-default feature flags, where setting those flags to false uses the new behaviour - that way, people upgrading don't have to change anything right away, but those who want to use the new features can disable the old behaviour on a case-by-case basis, and if they do so, they don't have to change anything going forward (when the flags are removed, they would just be ignored, no reason to warn or raise an error).

Thoughts?