Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save thangchung/6800c17e5cba4487165e30eb4c868b0e to your computer and use it in GitHub Desktop.
Save thangchung/6800c17e5cba4487165e30eb4c868b0e to your computer and use it in GitHub Desktop.
Using Lima to run containers with containerd and nerdctl (without Docker Desktop) on M1 Macs

Used M1 Mac mini 2020 with macOS Big Sur Version 11.5.2.

1. Install Patched QEMU

According to the GitHub issue comment in Lima's GitHub repository, Lima requires a patched QEMU on M1 Macs.

Note that the followings are customized steps by @toricls based on the original steps and the script. Be sure to check the original files when you try it on your own.

Install QEMU on Silicon based Apple Macs

# Install necessary packages for building
brew install libffi gettext glib pkg-config autoconf automake pixman ninja

# Clone qemu using ghq instead of git
ghq get https://github.com/qemu/qemu

# Change directory to qemu repository
# Note: In my case, git repositories are stored under $HOME/go/src directory
cd $HOME/go/src/$(ghq list | grep qemu)

# Checkout to commit dated June 03, 2021 v6.0.0
git checkout 3c93dfa42c394fdd55684f2fbf24cf2f39b97d47

# Apply patch series v8 by Alexander Graf
curl https://patchwork.kernel.org/series/485309/mbox/ | git am

# Building qemu installer
mkdir build && cd build
../configure --target-list=aarch64-softmmu
make -j8

# Install qemu
sudo make install

Create Ubuntu Server using QEMU on Silicon based Apple Macs

"First run, close manually after installation aka first reboot." steps

# Change directory to qemu repository
# Note: Git repositories are stored under $HOME/go/src directory in my case
cd $HOME/go/src/$(ghq list | grep qemu)

mkdir /tmp/test-patched-qemu && cd /tmp/test-patched-qemu

curl https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.2-live-server-arm64.iso -o ubuntu-lts.iso

qemu-img create -f qcow2 virtual-disk.qcow2 8G

cp $(dirname $(which qemu-img))/../share/qemu/edk2-aarch64-code.fd .

dd if=/dev/zero conv=sync bs=1m count=64 of=ovmf_vars.fd

# you'll see four files in the /tmp/test-patched-qemu directory, 1) edk2-aarch64-code.fd, 2) ovmf_vars.fd, 3) ubuntu-lts.iso, and 4) virtual-disk.qcow2
ls

qemu-system-aarch64 \
  -machine virt,accel=hvf,highmem=off \
  -cpu cortex-a72 -smp 4 -m 4G \
  -device virtio-gpu-pci \
  -device virtio-keyboard-pci \
  -drive "format=raw,file=edk2-aarch64-code.fd,if=pflash,readonly=on" \
  -drive "format=raw,file=ovmf_vars.fd,if=pflash" \
  -drive "format=qcow2,file=virtual-disk.qcow2" \
  -cdrom ubuntu-lts.iso

# I just tested just starting up it without any issues and skipped the "Second run, enjoy your Ubuntu Server." step, but you can of course finish instalation steps youself for your newly created VM to interact with it

2. Play with Lima

Installation

# According to the [installation doc](https://github.com/lima-vm/lima#installation), Lima supports Homebrew only for Intel Macs.

# Installing Lima manually
mkdir /tmp/lima && cd /tmp/lima

curl -L https://github.com/lima-vm/lima/releases/download/v0.6.1/lima-0.6.1-Darwin-arm64.tar.gz -o ./lima-0.6.1-Darwin-arm64.tar.gz

# List content in the tar.gz file
tar tzf /tmp/lima/lima-0.6.1-Darwin-arm64.tar.gz

# Extract them
tar -xzvf /tmp/lima/lima-0.6.1-Darwin-arm64.tar.gz -C /tmp/lima

# Rename the command "nerdctl.lima" to "nerdctl"
mv /tmp/lima/bin/nerdctl.lima /tmp/lima/bin/nerdctl

# Move the commands
sudo mv ./bin/* /usr/local/bin/
sudo mv ./share/lima /usr/local/share

Start VM

Save the following as default.yaml. It's altered from the original file for 1) setting loadDotSSHPubKeys to false, 2) removing the "mounts" section, and 3) changing num of cpu and disk size.

# ===================================================================== #
# BASIC CONFIGURATION
# ===================================================================== #

# Arch: "default", "x86_64", "aarch64".
# "default" corresponds to the host architecture.
arch: "default"

# An image must support systemd and cloud-init.
# Ubuntu and Fedora are known to work.
# Default: none (must be specified)
images:
  # Try to use a local image first.
  - location: "~/Downloads/hirsute-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "~/Downloads/hirsute-server-cloudimg-arm64.img"
    arch: "aarch64"

  # Download the file from the internet when the local file is missing.
  # Hint: run `limactl prune` to invalidate the "current" cache
  - location: "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img"
    arch: "aarch64"

# CPUs: if you see performance issues, try limiting cpus to 1.
# Default: 4
cpus: 2

# Memory size
# Default: "4GiB"
memory: "4GiB"

# Disk size
# Default: "100GiB"
disk: "30GiB"

ssh:
  # A localhost port of the host. Forwarded to port 22 of the guest.
  # Currently, this port number has to be specified manually.
  # Default: none
  localPort: 60022
  # Load ~/.ssh/*.pub in addition to $LIMA_HOME/_config/user.pub .
  # This option is useful when you want to use other SSH-based
  # applications such as rsync with the Lima instance.
  # If you have an insecure key under ~/.ssh, do not use this option.
  # Default: true
  loadDotSSHPubKeys: false

# ===================================================================== #
# ADVANCED CONFIGURATION
# ===================================================================== #

containerd:
  # Enable system-wide (aka rootful)  containerd and its dependencies (BuildKit, Stargz Snapshotter)
  # Default: false
  system: false
  # Enable user-scoped (aka rootless) containerd and its dependencies
  # Default: true
  user: true

# ===================================================================== #
# FURTHER ADVANCED CONFIGURATION
# ===================================================================== #

firmware:
  # Use legacy BIOS instead of UEFI.
  # Default: false
  legacyBIOS: false

video:
  # QEMU display, e.g., "none", "cocoa", "sdl".
  # As of QEMU v5.2, enabling this is known to have negative impact
  # on performance on macOS hosts: https://gitlab.com/qemu-project/qemu/-/issues/334
  # Default: "none"
  display: "none"

network:
  # The instance can get routable IP addresses from the vmnet framework using
  # https://github.com/lima-vm/vde_vmnet. Both vde_switch and vde_vmnet
  # daemons must be running before the instance is started. The interface type
  # (host, shared, or bridged) is configured in vde_vmnet and not lima.
  vde:
    # vnl (virtual network locator) points to the vde_switch socket directory,
    # optionally with vde:// prefix
    # - vnl: "vde:///var/run/vde.ctl"
    #   # VDE Switch port number (not TCP/UDP port number). Set to 65535 for PTP mode.
    #   # Default: 0
    #   switchPort: 0
    #   # MAC address of the instance; lima will pick one based on the instance name,
    #   # so DHCP assigned ip addresses should remain constant over instance restarts.
    #   macAddress: ""
    #   # Interface name, defaults to "vde0", "vde1", etc.
    #   name: ""

# Port forwarding rules. Forwarding between ports 22 and ssh.localPort cannot be overridden.
# Rules are checked sequentially until the first one matches.
# portForwards:
#   - guestPort: 443
#     hostIP: "0.0.0.0" # overrides the default value "127.0.0.1"; allows privileged port forwarding
#   # default: hostPort: 443 (same as guestPort)
#   # default: guestIP: "127.0.0.1" (also matches bind addresses "0.0.0.0", "::", and "::1")
#   # default: proto: "tcp" (only valid value right now)
#   - guestPortRange: [4000, 4999]
#     hostIP:  "0.0.0.0" # overrides the default value "127.0.0.1"
#   # default: hostPortRange: [4000, 4999] (must specify same number of ports as guestPortRange)
#   - guestPort: 80
#     hostPort: 8080 # overrides the default value 80
#   - guestIP: "127.0.0.2" # overrides the default value "127.0.0.1"
#     hostIP: "127.0.0.2" # overrides the default value "127.0.0.1"
#   # default: guestPortRange: [1024, 65535]
#   # default: hostPortRange: [1024, 65535]
#   - guestPort: 8888
#     ignore: true (don't forward this port)
#   # Lima internally appends this fallback rule at the end:
#   - guestIP: "127.0.0.1"
#     guestPortRange: [1024, 65535]
#     hostIP: "127.0.0.1"
#     hostPortRange: [1024, 65535]
#   # Any port still not matched by a rule will not be forwarded (ignored)

# ===================================================================== #
# END OF TEMPLATE
# ===================================================================== #
$ limactl start default.yaml
? Creating an instance "default" Proceed with the default configuration
INFO[0001] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.11.1/nerdctl-full-0.11.1-linux-arm64.tar.gz" (sha256:e2c8d0417b2fb79919f22a818813c646ad7ce0e600a951b6bac98340650e4435)
INFO[0001] Using cache "/Users/xxxx/Library/Caches/lima/download/by-url-sha256/cf356d64aa826ad177396feef432a14e0df5b2bb998191eff281541d329ff12c/data"
INFO[0001] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-arm64.img"
INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img"
INFO[0002] Using cache "/Users/xxxx/Library/Caches/lima/download/by-url-sha256/c40afd2acd5f2078759e01e119168213674615fff0701d53bc2bf69e5be3bf69/data"
INFO[0002] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/xxxx/.lima/default/serial.log")
INFO[0002] SSH Local Port: 60022
INFO[0002] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0012] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0019] [hostagent] The essential requirement 1 of 2 is satisfied
INFO[0019] [hostagent] Waiting for the essential requirement 2 of 2: "the guest agent to be running"
INFO[0022] [hostagent] The essential requirement 2 of 2 is satisfied
INFO[0022] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available"
INFO[0022] [hostagent] Forwarding "/run/user/505/lima-guestagent.sock" (guest) to "/Users/xxxx/.lima/default/ga.sock" (host)
INFO[0022] [hostagent] The optional requirement 1 of 2 is satisfied
INFO[0022] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed"
INFO[0022] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0022] [hostagent] Not forwarding TCP 0.0.0.0:22
INFO[0022] [hostagent] Not forwarding TCP [::]:22
INFO[0037] [hostagent] The optional requirement 2 of 2 is satisfied
INFO[0037] READY. Run `lima` to open the shell.

Run nginx container inside Lima VM

$ lima nerdctl run -d --name nginx -p 127.0.0.1:8080:80 nginx:alpine

docker.io/library/nginx:alpine:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:859ec6f2dc548cd2e5144b7856f2b5c37b23bd061c0c93cfa41fb5fb78307ead:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:4375f141ad62ca2763d3698e81ab6c61fa2cb8dba169212bf92c845158dbafb1: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:78754800625ea92fe7b32be0754194394e84a2ec0018b037f4279822bfcf5712:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0a8cd3c047c9c865bf65fa310e119ccbd3264a4050c81d0d6b5c3fd153dcce30:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:552d1f2373af9bfe12033568ebbfb0ccbb0de11279f9a415a29207e264d7f4d9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:50f60b67a614a5a3025f6d389391af19154139b35d314b2aa89dcd2791c4050b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e14bcbebf464b5696ca521435ef24348933f399f49bc4c3f4bce4a6c9b6fce5b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d8e1f9d1b87e92444b4e8e5dab10322aa2d66c1a226e1419048328a36fc0f716:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4d5947aec77d85cbbdc91c6bf3435095c91d516cf90af8d1394d1538d0103dcd:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.1 s                                                                    total:  9.3 Mi (1.8 MiB/s)
ab3ff970790da2ca30a7007b80e1fb65bbe86a4e39809096f764ec8c4cdbdba7

$ curl -I http://localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.21.1
Date: Wed, 01 Sep 2021 04:23:20 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 06 Jul 2021 15:21:34 GMT
Connection: keep-alive
ETag: "60e474fe-264"
Accept-Ranges: bytes

Use alias docker for lima nerdctl

$ lima nerdctl ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED           STATUS    PORTS                     NAMES
ab3ff970790d    docker.io/library/nginx:alpine    "/docker-entrypoint.…"    39 seconds ago    Up        127.0.0.1:8080->80/tcp    nginx

$ alias docker="lima nerdctl"

$ docker ps
CONTAINER ID    IMAGE                             COMMAND                   CREATED               STATUS    PORTS                     NAMES
ab3ff970790d    docker.io/library/nginx:alpine    "/docker-entrypoint.…"    About a minute ago    Up        127.0.0.1:8080->80/tcp    nginx

Woot! Woot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment