How to use lima as an alternative to Docker Desktop for mac

docker.yaml

# nikvdp: adapted from [1] to use rootfull docker and make homedir writeable
# [1]: https://github.com/lima-vm/lima/blob/master/examples/docker.yaml

# To make the linux docker socket accessible in macOS use this ssh command
# $ ssh -f -N -p 60006 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes -L $HOME/docker.sock:/var/run/docker.sock 127.0.0.1
# $ export DOCKER_HOST=unix://$HOME/docker.sock
# $ docker ...

images:
  # Hint: run `limactl prune` to invalidate the "current" cache
  - location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img"
    arch: "aarch64"
mounts:
  - location: "~"
    writable: true
  - location: "/tmp/lima"
    writable: true
ssh:
  localPort: 60006
  # Load ~/.ssh/*.pub in addition to $LIMA_HOME/_config/user.pub , for allowing DOCKER_HOST=ssh:// .
  # This option is enabled by default.
  # If you have an insecure key under ~/.ssh, do not use this option.
  loadDotSSHPubKeys: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
  - mode: system
    script: |
      #!/bin/bash
      set -eux -o pipefail
      command -v docker >/dev/null 2>&1 && exit 0
      export DEBIAN_FRONTEND=noninteractive
      curl -fsSL https://get.docker.com | sh
probes:
  - script: |
      #!/bin/bash
      set -eux -o pipefail
      if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
        echo >&2 "docker is not installed yet"
        exit 1
      fi
    hint: See "/var/log/cloud-init-output.log". in the guest

explanation.md

How to use lima as an alternative to Docker Desktop for mac

originally posted here: https://twitter.com/ArghZero/status/1455376756530036736

finally found a good nag-free alternative to Docker Desktop for m1 macs! (spoiler: it's lima https://github.com/lima-vm/lima).

everything i want out of mac docker: runs containers with the standard docker cli, docker-compose works as expected, and port forwards and shared vols do too!

getting it to work takes a bit of tinkering, here's how to do it:

install lima (brew install lima), then set up a config. lima uses cloud-init yamls to configure its vms. there's a docker example, but it uses rootless and sets shares readonly, which caused issues for me.

this modified version of their example fixes those issues: https://gist.github.com/nikvdp/78e8c6746b22e6145df03d3085b13ead#file-docker-yaml

download it and save it somewhere, then have lima fire up qemu:

limactl start ./docker.yaml

this may take a while on the first run because it needs to download a vm image and install docker into it. but once it's done you'll have a docker-capable (aarch64) linux vm running on your mac!

now we just need to configure it...

the first step is to add your user on the vm to the linux group and reboot the vm:

limactl shell docker sudo gpasswd -a $(whoami) docker
limactl stop docker
limactl start docker

once the vm is back up, you can shell in and make sure docker works from inside the vm. an easy way to test is to run this:

limactl shell docker docker ps

if you don't get any error msgs docker works on the linux side.

now we just need to tell macOS docker tools how to use it

docker is based on a client-server architecture, so what we're actually doing here is telling the mac docker client how to reach the docker server running in our new linux vm. docker normally uses unix sockets to communicate. luckily macOS supports these, and ssh can tunnel them!

this monster of a cmd creates a ~/docker.sock "file" on your mac that tunnels cmds issued to ~/docker.sock to the real docker.sock in the vm:

ssh -f -N -p 60006 -i ~/.lima/_config/user -o NoHostAuthenticationForLocalhost=yes -L "$HOME/docker.sock:/var/run/docker.sock" 127.0.0.1

once that's in place, we just need to tell macos docker commands to use it. this requires setting the DOCKER_HOST env var like so:

export DOCKER_HOST="unix:///$HOME/docker.sock"

with that done you should be able to run docker ps on your mac and get the usual output.

so at this point you technically have docker working on your mac, but it's kinda of a pain to use since you have to run that monstrosity of an ssh command and the export DOCKER_HOST command from each term where you want to use docker.

next step, make it convenient! 👇

you can drop this (hacktastic) fn into your ~/.zshrc or ~/.bashrc and call it. it will bring up the ssh tunnel if needed and set DOCKER_HOST correctly, yielding something pretty close to the old docker desktop for mac experience, minus the update prompts! https://gist.github.com/nikvdp/78e8c6746b22e6145df03d3085b13ead#file-lima-connect-sh

one caveat: if you use any GUI-based docker tools they probably won't automatically pick up the DOCKER_HOST variable. you can often still get them to work by doing echo $DOCKER_HOST | pbcopy and pasting the output into the app's cfg tho!

happy nag-free dockering!

lima-connect.sh

# attempt to unobtrusively create an ssh tunnel to a lima vm's docker socket
# and set DOCKER_HOST to use it if pasting into your ~/.bashrc or ~/.zshrc
# remember to call the fn by putting `lima-connect-docker` somewhere below
# where this fn is pasted in!
lima-connect-docker() {
    [[ "$(uname)" -ne "Darwin" ]] && return 0
    local lima_socket_path="/tmp/lima-docker.sock"

    lima-check-docker-socket-up() {
        local DOCKER_HOST
        export DOCKER_HOST="unix://$lima_socket_path" 
        ps ax | grep ssh | grep lima | grep -q "$lima_socket_path" && docker ps &> /dev/null 
    }

    lima-check-up() {
        ps ax | grep limactl | grep -q docker  
    }

    lima-start-socket-tunnel() {
        ssh -f -N -p 60006 -i ~/.lima/_config/user \
            -o NoHostAuthenticationForLocalhost=yes \
            -L "$lima_socket_path:/var/run/docker.sock" \
            127.0.0.1
    }

    if ! lima-check-docker-socket-up; then 
        ps ax | grep ssh | grep lima | grep docker.sock | awk '{print $1}' | xargs kill
        rm "$lima_socket_path"
        lima-start-socket-tunnel
    fi

    if lima-check-up && lima-check-docker-socket-up && [[ -r ~/docker.sock ]]; then 
        export DOCKER_HOST="unix://$lima_socket_path"
    fi

    # clean up
    unset -f lima-check-docker-socket-up 
    unset -f lima-check-up
    unset -f lima-start-socket-tunnel
}