Docker migrate to overlay2 from aufs script

_docker-migrate-aufs.md

Docker migrate to overlay2 from aufs script

Crazy that this is pretty much forced change without proper transition script

note. Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockage-aufs-vers-overlay2/ NOT WORKING!!!! IF FOLLOWING THE ABOVE.. SOMEHOW THE CONTAINERS DO NOT RE-APPEAR!

The only way I found that is somewhat automated is the the docker-compose way..

Which is still not 100% and require manual fixing of stupid errors of the docker-compose tool (mainly things that ere not interpreted right, like dates & userIds that need to manually surrounded by quotes etc)

Also I needed to add network_mode: bridge or network_mode: host to the configs as I did not want a network definition for each container which seems to be the docker-compose default way

Backup the whole crap before doing heart surgery

sudo systemctl stop docker
sudo cp -au /var/lib/docker /var/lib/docker.bk
sudo systemctl start docker

Export Containers

Export all container configs

cd ~
mkdir ~/dockercontainers
sudo docker container list -a > ~/dockercontainers/containers.txt
cat  ~/dockercontainers/containers.txt

CIDS=$(docker container list -a | sed '1d' | awk '{print $1}')
for c in $CIDS; do mkdir ~/dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > ~/dockercontainers/$c/docker-compose.yml ; done

# or if local (with fixes)
for c in $CIDS; do mkdir ~/dockercontainers/$c ; ~/autocompose.py $c > dockercontainers/$c/docker-compose.yml ; done

# check if there are no errors in the created files
for c in $CIDS; do cd ~/dockercontainers/$c ; echo ======== $c ======= ;docker-compose config  ; done

#Fix possible errors... e.g. adding quotes around strings docker-compose does not swallow (e.g. dates)

Store all images (I think not needed if no persistent references are used)

cd ~
mkdir dockersave
sudo docker images > dockersave/list.txt
cat  dockersave/list.txt

IDS=$(docker images | sed '1d' | awk '{print $3}')
for c in $IDS; do docker save -o dockersave/$c.tar $c; done
cat dockersave/list.txt | sed '1d' | grep -v "" | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag

Change driver

sudo systemctl stop docker
sudo nano  /etc/docker/daemon.json

if the file is already there, add line "storage-driver" : "overlay2"

if the file is empty, you can go with echo '{ "storage-driver" : "overlay2" }' > /etc/docker/daemon.json sudo systemctl start docker

What also works is edit the file first and than just restart (so no need to stop first) sudo systemctl restart docker

Restore images

cd dockersave/
IDS=$(ls *.tar)
for c in $IDS; do docker load -i $c; done

Restore containers

for c in $CIDS; do cd ~/dockercontainers/$c ; docker-compose up -d --no-deps --build ; done

When something goes horrably wrong you can remove the containers and try again

docker container stop `docker container ls -aq`
docker container rm `docker container ls -aq`

autocompose.py

#! /usr/bin/env python

# Copy of https://github.com/Red5d/docker-autocompose dirty hacked with some fixes for dates and 
# possiblity to create docker command lines from the existing containers by using the -o2 parameter

import sys, argparse, pyaml, docker
from collections import OrderedDict

def main():
    parser = argparse.ArgumentParser(description='Generate docker-compose yaml definition from running container.')
    parser.add_argument('-v', '--version', type=int, default=3, help='Compose file version (1 or 3)')
    parser.add_argument('-o', '--output', type=int, default=0, help='Type of output (0 or 1,2)')
    parser.add_argument('cnames', nargs='*', type=str, help='The name of the container to process.')
    args = parser.parse_args()

    struct = {}
    networks = []
    for cname in args.cnames:
        cfile, networks,cmdline = generate(cname)
        struct.update(cfile)

    if args.output >0:
       print ("\r\n"+ cmdline + "\r\n")
    if args.output <2:
       render(struct, args, networks)


def render(struct, args, networks):
    # Render yaml file
    if args.version == 1:
        pyaml.p(OrderedDict(struct))
    else:
        pyaml.p(OrderedDict({'version': '"3"', 'services': struct, 'networks': networks}))

def generate(cname):
    c = docker.from_env()

    try:
        cid = [x.short_id for x in c.containers.list(all=True) if cname == x.name or x.short_id in cname][0]
    except IndexError:
        print("That container is not available.")
        sys.exit(1)

    cattrs = c.containers.get(cid).attrs


    # Build yaml dict structure

    cfile = {}
    cfile[cattrs['Name'][1:]] = {}
    ct = cfile[cattrs['Name'][1:]]
    cmdLine ="docker run --name  " + str( cattrs['Name'][1:])

    values = {
        'cap_add': cattrs['HostConfig']['CapAdd'],
        'cap_drop': cattrs['HostConfig']['CapDrop'],
        'cgroup_parent': cattrs['HostConfig']['CgroupParent'],
        'container_name': cattrs['Name'][1:],
        'devices': [],
        'dns': cattrs['HostConfig']['Dns'],
        'dns_search': cattrs['HostConfig']['DnsSearch'],
        'environment': cattrs['Config']['Env'],
        'extra_hosts': cattrs['HostConfig']['ExtraHosts'],
        'image': cattrs['Config']['Image'],
        'labels': cattrs['Config']['Labels'],
        'links': cattrs['HostConfig']['Links'],
        #'log_driver': cattrs['HostConfig']['LogConfig']['Type'],
        #'log_opt': cattrs['HostConfig']['LogConfig']['Config'],
        #'logging': {'driver': cattrs['HostConfig']['LogConfig']['Type'], 'options': cattrs['HostConfig']['LogConfig']['Config']},
        'networks': {x for x in cattrs['NetworkSettings']['Networks'].keys() if x != 'bridge'},
        'security_opt': cattrs['HostConfig']['SecurityOpt'],
        'ulimits': cattrs['HostConfig']['Ulimits'],
        'volumes': cattrs['HostConfig']['Binds'],
        'volume_driver': cattrs['HostConfig']['VolumeDriver'],
        'volumes_from': cattrs['HostConfig']['VolumesFrom'],
        'entrypoint': cattrs['Config']['Entrypoint'],
        'user': cattrs['Config']['User'],
        'working_dir': cattrs['Config']['WorkingDir'],
        'domainname': cattrs['Config']['Domainname'],
        'hostname': cattrs['Config']['Hostname'],
        'ipc': cattrs['HostConfig']['IpcMode'],
        'mac_address': cattrs['NetworkSettings']['MacAddress'],
        'privileged': cattrs['HostConfig']['Privileged'],
        'restart': cattrs['HostConfig']['RestartPolicy']['Name'],
        'read_only': cattrs['HostConfig']['ReadonlyRootfs'],
        'stdin_open': cattrs['Config']['OpenStdin'],
        'tty': cattrs['Config']['Tty']
    }

    # Populate devices key if device values are present
    if cattrs['HostConfig']['Devices']:
        values['devices'] = [x['PathOnHost']+':'+x['PathInContainer'] for x in cattrs['HostConfig']['Devices']]

    networks = {}
    if values['networks'] == set():
        del values['networks']
        values['network_mode'] = 'bridge'
    else:
        networklist = c.networks.list()
        for network in networklist:
            if network.attrs['Name'] in values['networks']:
                networks[network.attrs['Name']] = {'external': (not network.attrs['Internal'])}

    # Check for command and add it if present.
    if cattrs['Config']['Cmd'] != None:
        values['command'] = " ".join(cattrs['Config']['Cmd']),

    # Check for exposed/bound ports and add them if needed.
    try:
        expose_value =  list(cattrs['Config']['ExposedPorts'].keys())
        ports_value = [cattrs['HostConfig']['PortBindings'][key][0]['HostIp']+':'+cattrs['HostConfig']['PortBindings'][key][0]['HostPort']+':'+key for key in cattrs['HostConfig']['PortBindings']]

        # If bound ports found, don't use the 'expose' value.
        if (ports_value != None) and (ports_value != "") and (ports_value != []) and (ports_value != 'null') and (ports_value != {}) and (ports_value != "default") and (ports_value != 0) and (ports_value !=$
            for index, port in enumerate(ports_value):
                if port[0] == ':':
                    ports_value[index] = port[1:]
#                print ( "-p " , str (ports_value[index]))
                cmdLine = cmdLine + " -p " + str (ports_value[index])
            values['ports'] = ports_value
        else:
            values['expose'] = expose_value

    except (KeyError, TypeError):
        # No ports exposed/bound. Continue without them.
        ports = None

    # fix some dates from causing errors.
    try:
        if values['labels']['org.label-schema.build-date'] != None:
          d =  values['labels']['org.label-schema.build-date']
          values['labels']['org.label-schema.build-date']  = "'" + str(  d ) +  "'"
    except KeyError as ke:
      pass

    # Iterate through values to finish building yaml dict.
    for key in values:
        value = values[key]
        if (value != None) and (value != "") and (value != []) and (value != 'null') and (value != {}) and (value != "default") and (value != 0) and (value != ",") and (value != "no"):
            ct[key] = value

    if cattrs['HostConfig']['Binds']:
     for key in  cattrs['HostConfig']['Binds']:
       cmdLine = cmdLine + ' -v ' + str(key)

    if cattrs['Config']['Env']:
     for key in values['environment']:
         cmdLine = cmdLine + ' -e ' + str(key.split('=')[0])
         if len(key.split('='))>1:
            cmdLine = cmdLine + '="' +  str(key.split('=')[1])  + '"'
    cmdLine =  cmdLine +  " " + str ( cattrs['Config']['Image'])
    return cfile, networks, cmdLine


if __name__ == "__main__":
    main()

Great work, thank you! :)

Great work, but there are some issues:

Line 114 of the above autocompose.py ends with:

 and (ports_value !=$

That's a syntax error. I have changed that to

  and (ports_value != 'no'):

to match the original IGNORE_VALUES.

Also need to pin package requests<2.29. See docker/docker-py#3113

Whenever a named volume was used, there was an error in the generated docker-compose.yml files:

• Named volume used, but no declaration found

Besides of that, instead of using sed and awk to tweak docker's output I would suggest letting docker do the work itself by creating the list of container IDs with

docker container list -a --format '{{.ID}}'

Similarly, you can obtain the list of image IDs directly with

docker image ls --format '{{.ID}}' 

However, there can be duplicates, if the same image exists with several tags (for example latest and an explicit version tag). In that case, the duplicates should be eliminated by piping the output through

sort -u

That applies to the original version as well.

Instead of IDs, the autocompose.py script also accepts a container name. A list of container names can be obtained with

docker container list -a --format '{{.Names}}'

This allows to organize the generated docker-compose files in folders named after the container name. Whether or not this is better understandable than the container ids depends on the names given when running the container. If one has mostly the autogenerated names à la affectionate_newton then may be the ID is better. YMMV.

I really appreciate having access to this gist! Thank you.

I'm still working through a couple of issues, and would appreciate any help/expertise.

1. Tagging of the restore images didn't work for me. First, the tag file is empty unless I remove grep -v "" from the command to create the tag file. I do not understand what that was supposed to do?
2. I also had to execute tag to then add the tags and repo sources to the images. No issue once I figured out that, but maybe this note will help someone else out. Or, does this mean something else isn't correct?
3. Most of our original containers had been created with docker-compose. For those, is it necessary to use the the created docker-compose.yml files on the first run? I did, but then had one container not start (turned out I was out of space again) and so I nuked them. All but one of our containers starts with a single compose file, so I just used that and the generated file to kick start the one app. It seems to be working, but not sure if I'm misunderstanding something here?

BTW, this process took a LOT of storage space. I had to attach a second volume for holding the images, backup and other files.

3. Most of our original containers had been created with docker-compose.
   For those, is it necessary to use the the created docker-compose.yml files on the first run?

The purpose of the generated docker-compose.yml files is to be able to re-create the containers using the new storage driver. As @marcelrv gives the instruction "# Fix possible errors", that means that the generated files can have errors which you would need to fix. If you already have a working docker-compose.yml then IMHO it is always the better choice to use it. I have successfully migrated all my containers and this gist was a great help. But I have used it only for everything that was created "by hand". For services already defined with docker-compose.yml files, I just continue to use them with the new storage driver. That does work.

BTW, this process took a LOT of storage space. I had to attach a second volume for holding the images, backup and other files.

Yes. IMHO using a separate drive for the backup is a good idea anyway. I have even used a second computer for a dry run before I scratched everything on the "master" server.

For services already defined with docker-compose.yml files, I just continue to use them with the new storage driver. That does work.

Hahahaha! Wow. I wish I had realized that earlier. Would've saved me a lot of time :-)

Thank you for reply. I have to do one more server, and I'm pretty sure all of the containers are from existing compose files.

Try... Catch... Finally... Eventually...

I came to working steps / script...

Follow the above scenario, but this works, without grep

cat dockersave/list.txt | sed '1d' | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag
and, after a step to load images, you have to run

bash dockersave/tag

Which is not mentioned anywhere above...

Also, if you have old version of docker-compose you should

cd /usr/local/bin && sudo rm -rf docker-compose

and then download newest docker-compose, place it to /usr/local/bin and chmod +x it
You cam get it from here: https://github.com/docker/compose/releases
Also don't forget to remove backed-up directory under /var/lib/docker.bk

Also after this, you might want/need to completley remove docker, but don't delete/purge images and config files, and then upgrade docker...

We were able to sucessfully upgrade from ubuntu 14 to ubuntu 22 :) what a jump...

Please correct me if I'm wrong, but...

The steps that appear to have worked for me were no more complicated than:

• sudo systemctl stop docker.service;
• Edit /etc/docker/daemon.json to include the "storage-driver" : "overlay2" tuple;
• sudo systemctl start docker.service;

After this:

• journalctl -u docker.service --since '10 minutes ago' no longer reported warnings about using aufs, like:

  ... dockerd[...]: time="..." level=info msg="[graphdriver] using prior storage driver: aufs"
  ... dockerd[...]: time="..." level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
  ... dockerd[...]: time="..." level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=aufs version=20.10.7
  

  ... and instead, reports:

  ... dockerd[...]: time="..." level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=overlay2 version=20.10.7
  

• Morever, both /var/lib/docker/ and /var/lib/docker/image/ now each have overlay2/ sub-directories that they didn't have before; (their repective aufs/ sub-directories are still there, too, btw);
• docker image ls, docker container ls and docker service ls all report the same data as they did before...
  ... except for:
  • docker image ls which now list <none> for tags;
• Not to mention that the deployed stack also still appears to work as before;

Do note that this all is on a host in swarm mode (i.e., it had been docker swarm init ... and docker stack deploy --compose-file docker-compose.yml ...-ed at some point);

It would appear that all images were automatically converted from aufs to overlay2 by this, except for the replication of their tags, and hence that the only remaining housekeeping expected to-do, is to get those reattached.

Additionally, I think I should rm -r /var/lib/docker{,/image}/aufs once truly convinced that they're no longer needed.

For restoring and tagging the images on the same step, a restore script can be created as follows:

cat tag | awk '{ OFS="";print "docker load < ",$3, ".tar; ", $0}' > restore_images.sh

This assumes that tag contents look like this:

#tag contents
docker tag e34e831650c1 ubuntu:latest
docker tag f3d89a2abe0d postgres:15

The restore_images.sh script will look like:

docker load < e34e831650c1.tar; docker tag e34e831650c1 ubuntu:latest
docker load < f3d89a2abe0d.tar; docker tag f3d89a2abe0d postgres:15

Then run bash restore_images.sh.