IOTstack and "override" files

IOTstackOverrideFiles.md

IOTstack and "override" files

As a general principle, I'm not in favour of override files. I think they muddy the waters. I think you're much better off editing docker-compose.yml directly.

That said…

When you use IOTstack to manage Docker containers, you have access to two override mechanisms:

1. A compose-override.yml file. This mechanism is peculiar to IOTstack and is implemented by the so-called "new" menu on the master branch.
2. A docker-compose.override.yml file. This mechanism is part of docker-compose.

Of the two, the second mechanism is easier to use.

Reference files

Assume the following files are present and used throughout this gist.

example compose file

1. Path:

   ~/IOTstack/docker-compose.yml
   

2. Content:

   version: '3.6'
   
   services:
   
     example:
       container_name: example
       image: "example:latest"
       restart: unless-stopped
       environment:
         - TZ=Etc/UTC
         - EXAMPLE1="this is from the compose file"
         - EXAMPLE2="this is from the compose file"
       ports:
         - "1234:1234"
       volumes:
         - ./volumes/example/data:/var/lib/example

example override files

1. Two paths, one for each mechanism:

   ~/IOTstack/compose-override.yml
   ~/IOTstack/docker-compose.override.yml
   

2. Content (the two paths have identical content):

   services:
     example:
       environment:
       - TZ=Australia/ACT
       - EXAMPLE2="this is from the override file"
       - EXAMPLE3="this is from the override file"

The IOTstack mechanism

To use this mechanism, you:

1. Create (or edit) a file at the following path that contains your overrides:

   ~/IOTstack/compose-override.yml
   

2. Run the menu, and:

   • Choose "Build Stack".
   • Optionally – add and/or remove containers from the list.
   • Press return which "builds" the stack and applies the override file.
   • Choose "Exit".

example

The result of running the menu and applying the override to the reference files above is:

version: '3.6'
services:
  example:
    environment:
    - TZ=Australia/ACT
    - EXAMPLE2="this is from the override file"
    - EXAMPLE3="this is from the override file"
    container_name: example
    image: "example:latest"
    restart: unless-stopped
    ports:
    - "1234:1234"
    volumes:
    - ./volumes/example/data:/var/lib/example

Key points:

1. The EXAMPLE1 environment variable has been deleted. This is because this override mechanism works at the section level. In other words, if you decide to override one element in a section, you need to include all elements from that section. Any elements you omit from the override file will be deleted from the resulting compose file.

2. Elements in the compose file are re-ordered and re-aligned but the basic syntax of each element remains undisturbed.

3. Your compose file is overwritten by each menu run so any other edits will be lost.

4. The original (pre-override) version of the compose file is stored at:

   ~/IOTstack/services/docker-compose.save.yml
   

The docker-compose mechanism

To use this mechanism, you:

1. Create (or edit) a file at the following path:

   ~/IOTstack/docker-compose.override.yml
   

2. Run:

   $ cd ~/IOTstack
   $ docker-compose up -d

In other words, the override mechanism is automatic and occurs each time you bring up your stack or one of the containers. There is no separate step like running the menu.

To preview this mechanism, run:

$ cd ~/IOTstack
$ docker-compose config

example

The result of running the config command against the reference files given above is:

services:
  example:
    container_name: example
    environment:
      EXAMPLE1: '"this is from the compose file"'
      EXAMPLE2: '"this is from the override file"'
      EXAMPLE3: '"this is from the override file"'
      TZ: Australia/ACT
    image: example:latest
    ports:
    - published: 1234
      target: 1234
    restart: unless-stopped
    volumes:
    - /home/pi/IOTstack/volumes/example/data:/var/lib/example:rw
version: '3.6'

Key points:

1. The EXAMPLE1 environment variable has been retained. This is a true "merge" which works at the element level rather than the section level. However, it does imply that there is no way of using an override file to remove an element that is present in a compose file.
2. Elements are re-ordered and expressed in long-form syntax. However, you only ever see this if you run the config command. In normal use, everything happens silently behind the scenes when you run the up command.
3. Your compose file is never overwritten.

but wait, there's also a third mechanism

docker-compose can also take multiple -f arguments, each specifying a YAML file. The files are applied in the order in which they appear on the command line. Anything in a later YAML file implicitly overrides anything in an earlier file.

This scheme is more difficult to use because it is not integrated with the config command so there is no way to verify what is actually being implemented.