This tutorial presents a step by step guide on how to run docker containers directly on Android. By directly I mean there's no VM involved nor chrooting inside a GNU/Linux rootfs. This is docker purely in Android. Yes, it is possible.
Bear in mind that you'll have to root your phone, mess with and compile your phone's kernel and docker suit. So, be prepared to get your hands dirty.
This step is pretty device specific, so there's no way to write a generic tutorial here. You'll need to google for instructions for your device and follow them.
Just be aware that you may lose your phone's warrant and all your data will be erased after unlocking the bootloader, so make a backup of your important stuff.
Compiling the phone's kernel is also device specific, but some major tips may help you out.
First, google about instructions for your phone. Start by compiling the kernel without any modification. Flash it and hope for the best. If everything went well, then you can proceed to the modifications.
Note that flashing the kernel won't erase any data in your phone. The worst that can happen is you get stuck in a boot loop. In this case, you can flash a kernel that's known to be working or just flash a working ROM, since it contains a kernel with it. None of these operations erase any data in your phone.
Now that you (hopefully) are able to compile the kernel, let's talk about what matters. Docker needs a lot of features that are disabled by default in Android's kernel.
To check the necessary features list, first install the Termux app in your phone. This is the terminal emulator that we're going to use throughout this guide. It has a package manager with many tools compiled for Android.
Next, open Termux and download a script to check your kernel:
$ pkg install wget
$ wget https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh
$ chmod +x check-config.sh
$ sed -i '1s_.*_#!/data/data/com.termux/files/usr/bin/bash_' check-config.sh
$ sudo ./check-config.sh
Now, in your computer, open the kernel's configuration menu. This menu is a modified version of dialog, a ncurses window menu, in which you can enable and disable the kernel features. To look for some item in particular, you can press the / key and type the item name and hit Enter. This will show the description and location of the item.
For now, we want to enable the Generally Necessary items, the Network Drivers items and some Optional Features. For the Storage Drivers we'll be using the overlay.
Before compiling the kernel there are two files that need to be patched.
The first one is the kernel/Makefile. Although not strictly necessary to modify this file, it will help by making it possible to check if your kernel has all the necessary features docker needs.
If you do not apply this patch, the output of the check-config.sh script used above won't be reliable after recompiling the kernel.
Check the patch at the attachments section and modify your Makefile accordingly.
This second file needs to be patched because of a bug introduced by Google. After you run any container, a seg fault will be generated due to a null pointer dereference and your phone will freeze and reboot. If you work at Google or know someone who does, warn him/her about it.
Check the patch at the attachments section and modify your xt_qtaguid.c accordingly.
Now that everything is setup, compile and flash the kernel. If you applied the Makefile patch, you see this warning everytime your phone boots:
Don't worry though, this is a harmless warning remembering you that you're using a modified kernel.
Once you have a supported kernel, it's time to compile the docker suit. It's a suit because it's not just one program, but rather a set of different programs that we'll need to compile separately. So hands on.
Firts, let's install the packages we're gonna use to build docker in Termux:
$ pkg install go make
Now we're ready to start compiling things. Create a work directory where the packages will be downloaded and built:
$ mkdir $TMPDIR/docker-build
$ cd $TMPDIR/docker-build
Download all the patches files into there and let's begin. All commands for the differents packages that'll be compiled next is meant to be executed inside this folder.
This is the docker client, which will talk to the docker daemon. This package will compile a binary named docker and all docker man pages. To build and install it:
$ wget https://github.com/docker/cli/archive/v20.10.2.tar.gz -O cli-20.10.2.tar.gz
$ tar xf cli-20.10.2.tar.gz
$ mkdir -p src/github.com/docker
$ mv cli-20.10.2 src/github.com/docker/cli
$ export GOPATH=$(pwd)
$ export VERSION=v20.10.2-ce
$ export DISABLE_WARN_OUTSIDE_CONTAINER=1
$ cd src/github.com/docker/cli
$ xargs sed -i 's_/var/\(run/docker\.sock\)_/data/docker/\1_g' < <(grep -R /var/run/docker\.sock | cut -d':' -f1 | sort | uniq)
$ patch vendor/github.com/containerd/containerd/platforms/database.go ../../../../database.go.patch.txt
$ patch scripts/docs/generate-man.sh ../../../../generate-man.sh.patch.txt
$ patch man/md2man-all.sh ../../../../md2man-all.sh.patch.txt
$ patch cli/config/config.go ../../../../config.go.patch.txt
$ make dynbinary
$ make manpages
$ install -Dm 0755 build/docker-android-* $PREFIX/bin/docker
$ install -Dm 644 -t $PREFIX/share/man/man1 man/man1/*
$ install -Dm 644 -t $PREFIX/share/man/man5 man/man5/*
$ install -Dm 644 -t $PREFIX/share/man/man8 man/man8/*
The docker daemon is the most problematic binary that's gonna be compiled. It needs so many patches that's easier to modify the code in a batch with sed. Despite the need of modifying a lot of files, the modifications by themselfs are rather simple:
- Substitute every occurrence of
runtime.GOOSby the string"linux"; - Remove unneeded imports of the
runtimelib.
By doing that, we are in essence spoofing our operating system as a Linux one: everytime the code would do the runtime.GOOS == "linux" comparison (which would become "android" == "linux", and thus false) it will now do "linux" == "linux" and thus true.
To make the substitution across every file, we'll run a sed command. After that, some files will now give the extremely annoying unturnable-off go lang "feature" imported and not used error, because the only function these files were using from the runtime package was the runtime.GOOS. So, to fix it we'll use an horrible but simple solution: we'll keep trying to compile the code and at each failed attempt we'll fix the reported files till we get it to compile successfully.
$ wget https://github.com/moby/moby/archive/v20.10.2.tar.gz -O moby-20.10.2.tar.gz
$ tar xf moby-20.10.2.tar.gz
$ cd moby-20.10.2
$ export DOCKER_GITCOMMIT=8891c58a43
$ export DOCKER_BUILDTAGS='exclude_graphdriver_btrfs exclude_graphdriver_devicemapper exclude_graphdriver_quota selinux exclude_graphdriver_aufs'
$ xargs sed -i "s_\(/etc/docker\)_$PREFIX\1_g" < <(grep -R /etc/docker | cut -d':' -f1 | sort | uniq)
$ xargs sed -i 's/[a-zA-Z0-9]*\.GOOS/"linux"/g' < <(grep -R '[a-zA-Z0-9]*\.GOOS' | cut -d':' -f1 | sort | uniq)
$ while ! files=$(AUTO_GOPATH=1 PREFIX='' hack/make.sh dynbinary 2>&1 1>/dev/null); do xargs sed -i 's/\("runtime"\)/_ \1/' < <(echo $files | grep runtime | cut -d':' -f1 | cut -c38-); done
$ install -Dm 0755 bundles/dynbinary-daemon/dockerd-dev $PREFIX/bin
A binary called dockerd-dev was compiled and installed, but in order to it run correctly, the cgroups need to be mounted. Since Android mounts the cgroups in a non standard location we need to fix this. To do so, a script named dockerd will be created that will mount crgoups in the correct path if needed and call dockerd-dev next.
$ cat << "EOF" > $PREFIX/bin/dockerd
#!/data/data/com.termux/files/usr/bin/bash
export PATH="${PATH}:/system/xbin:/system/bin"
opts='rw,nosuid,nodev,noexec,relatime'
cgroups='blkio cpu cpuacct cpuset devices freezer memory pids schedtune'
# try to mount cgroup root dir and exit in.case of failure
if ! mountpoint -q /sys/fs/cgroup 2>/dev/null; then
mkdir -p /sys/fs/cgroup
mount -t tmpfs -o "${opts}" cgroup_root /sys/fs/cgroup || exit
fi
# try to mount cgroup2
if ! mountpoint -q /sys/fs/cgroup/cg2_bpf 2>/dev/null; then
mkdir -p /sys/fs/cgroup/cg2_bpf
mount -t cgroup2 -o "${opts}" cgroup2_root /sys/fs/cgroup/cg2_bpf
fi
# try to mount differents cgroups
for cg in ${cgroups}; do
if ! mountpoint -q "/sys/fs/cgroup/${cg}" 2>/dev/null; then
mkdir -p "/sys/fs/cgroup/${cg}"
mount -t cgroup -o "${opts},${cg}" "${cg}" "/sys/fs/cgroup/${cg}" \
|| rmdir "/sys/fs/cgroup/${cg}"
fi
done
# start the docker daemon
dockerd-dev $@
EOF
Make the script executable:
$ chmod +x $PREFIX/bin/dockerd
And finally configure some dockerd options:
$ mkdir -p $PREFIX/etc/docker
$ cat << "EOF" > $PREFIX/etc/docker/daemon.json
{
"data-root": "/data/docker/lib/docker",
"exec-root": "/data/docker/run/docker",
"pidfile": "/data/docker/run/docker.pid",
"hosts": [
"unix:///data/docker/run/docker.sock"
],
"storage-driver": "overlay2"
}
EOF
Warning: dockerd will store all its files, like containers, images, volumes, etc inside the /data/docker folder, which means you'll lose everything if you format the phone (flash a ROM). This folder was chosen instead of storing things inside Termux installation folder, because dockerd fails when setting up the overlay storage driver there. It seems Android mounts the /data/data folder with some options that prevent overlayfs to work, or the filesystem doesn't support it.
This is a dockerd dependency. Some patches are needed to fix path locations, build the manuals correctly and compile extra binaries used by dockerd that are not build by default by the Makefile:
$ wget https://github.com/containerd/containerd/archive/v1.4.3.tar.gz
$ tar xf v1.4.3.tar.gz
$ mkdir -p src/github.com/containerd
$ mv containerd-1.4.3 src/github.com/containerd/containerd
$ export GOPATH=$(pwd)
$ cd src/github.com/containerd/containerd
$ xargs sed -i "s_\(/etc/containerd\)_$PREFIX\1_g" < <(grep -R /etc/containerd | cut -d':' -f1 | sort | uniq)
$ patch runtime/v1/linux/bundle.go ../../../../bundle.go.patch.txt
$ patch runtime/v2/shim/util_unix.go ../../../../util_unix.go.patch.txt
$ patch Makefile ../../../../Makefile.patch.txt
$ patch platforms/database.go ../../../../database.go.patch.txt
$ patch vendor/github.com/cpuguy83/go-md2man/v2/md2man.go ../../../../md2man.go.patch.txt
$ BUILDTAGS=no_btrfs make -j8
$ make -j8 man
$ DESTDIR=$PREFIX make install
$ DESTDIR=$PREFIX/share make install-man
Lastly, some configurations:
$ mkdir -p $PREFIX/etc/containerd
$ cat << "EOF" > $PREFIX/etc/containerd/config.toml
root = "/data/docker/var/lib/containerd"
state = "/data/docker/run/containerd"
imports = ["$PREFIX/etc/containerd/runtime_*.toml", "./debug.toml"]
[grpc]
address = "/data/docker/run/containerd/containerd.sock"
[debug]
address = "/data/docker/run/containerd/debug.sock"
EOF
Note: unfortunately containerd files also can't be stored inside Termux installation folder, failing with an error when creating the socket it uses.
runc is a dependency of containerd. Conveniently for us, it's already provided by Termux's repository. Install it by simply:
$ pkg install runc
Now comes the truth time. To run the containers, first we need to start the daemon manually. To do so, it's advisable to install a terminal multiplexer so you can run the daemon in one pane and the container in others panes:
$ pkg install tmux
In one pane start dockerd:
$ sudo dockerd --iptables=false
And in others panes you can run the containers:
$ sudo docker run hello-world
Note: Teaching how to use tmux is out of the scope of this guide, you can find good tutorials on YouTube. If you don't wanna use a terminal multiplexer, you can run dockerd in the background instead, with dockerd &> /dev/null &.
After months testing docker, the only thing I couldn't managed to get working 100% as intended is the internet. I'm not sure why, but the only way to get the containers to access the internet is to use the --net=host flag. Trying to isolate the container network almost work: the tun tap interface is created, but it doesn't seem to work.
Yes, it's possible to run GUI programs inside a container! There's basically two ways of accomplishing it in a simple manner:
This method has the advantage of not making necessary the installation nor configuration of any additional programs inside the container; all you'll have to do is to setup the X inside termux and share its sockets with the container.
This is advisable to be used when you intend to run various containers with GUI, since you'll only have to install and setup a VNC once in the host, instead of doing it for each container. This will save storage space and time.
The first step is to enable the X11 repository in termux, this will allow installation of graphical interface related programs, like the VNC server we'll be using.
$ pkg install x11-repo
Then install a VNC server in termux:
$ pkg install tigervnc
Note: These installations steps need to be executed only once.
Now, just run it:
$ vncserver -noxstartup -localhost
Note: It's advisable to pass the -geometry HEIGHTxWEIGHT flag substituting HEIGHT and WEIGHT by your phone's screen resolution or some multiple of it.
Note: The very first time you run it, you'll be prompted to setup a password. Note that passwords are not visible when you are typing them and it's maximal length is 8 characters. If you don't wanna use a passwd, use the -SecurityTypes none flag.
If everything is okay, you will see this message:
New 'localhost:1 ()' desktop is localhost:1
It means that X (vnc) server is available on display 'localhost:1'. Finally, export the DISPLAY environment variable according to that value:
$ export DISPLAY=:1
Now that the VNC server is configured and running in the host, start the container sharing the X related files as volumes:
$ sudo docker run -ti \
--net="host" \
--dns="8.8.8.8" \
-e DISPLAY=$DISPLAY \
-v $TMPDIR/.X11-unix:/tmp/.X11-unix \
-v $HOME/.Xauthority:/root/.Xauthority \
ubuntu
Note: If by any reason you forget to export the DISPLAY before starting the container, you can still export it from inside it.
You'll now be able to launch GUI programs from inside the container, e.g.:
# echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf
# apt update
# apt install x11-apps
# xeyes
To check the GUI, you'll need to install a VNC client app in your Android phone, like VNC Viewer (developed by RealVNC Limited). Unfortunately it's not open source, but it's a good and intuitive VNC client for Android.
Note: There's also an open source alternative developed by @pelya called XServer XSDL, which will not be covered by this guide (for now).
After installing the VNC Viewer app, open it and setup a new connection using 127.0.0.1 (or localhost) as the IP, 5901 as the port (the port is calculated as 5900 + {display number}) and when/if prompted, type the password choosen when running vnctiger for the first time.
This method is very similar to the previous, with the difference that the X server will be installed inside the container instead of in the termux host.
The advantages are:
- you aren't changing your host system by installing softwares on it (like the VNC server);
- security, since you won't be sharing your host's X (this is only relevant when you are not the one running the container).
The main disadvantage is that you'll need to install and config the VNC server for each container you'd run a GUI program, thus making these containers big and time consuming to setup.
First, start a container:
$ sudo docker run -ti \
--net="host" \
--dns="8.8.8.8" \
ubuntu
Now, a VNC server needs to be installed and configured inside the container. You can choose between TigerVNC or x11vnc:
The same VNC server used above in termux. To install it:
# echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf
# apt update
# apt install tigervnc-standalone-server
Next, start it with:
# vncserver -noxstartup -localhost -SecurityTypes none
Here we disabled password (-SecurityTypes none) because using it causes things to crash as described in this issue report TigerVNC/tigervnc#800
If everything is okay, you will see this message:
New 'localhost:1 (root)' desktop at :1 on machine localhost
Export the DISPLAY environment variable according to that value:
# export DISPLAY=:1
From now on, you can already run GUI programs and access them using the VNC Viewer client as already described in the end of X11 Forwarding steps.
Install the x11vnc and the virtual fake X (since x11vnc can't emulate a X11 by itself):
# echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf
# apt update
# apt install x11vnc xvfb
Now, start it:
# x11vnc -nopw -forever -noshm -create
If everything is okay, you will see this message:
The VNC desktop is: localhost:0
PORT=5900
This will open a xterm terminal which can be acessed by the VNC Viewer client as already described in the end of X11 Forwarding steps. From that terminal you can open the desired GUI program.
I'm not talking about running the useless steam app for Android, but about running the Desktop version and play the games inside a docker container. Yes, you read it right, it's possible to play your Steam games on Android!
(ACTUALLY NOT YET, BECAUSE I DIDN'T MANAGE TO GET OPENGL TO WORK, THAT'S WHY THIS IS A WORK IN PROGRESS. TO CONTRIBUTE OR STAY UP TO DATE ABOUT THE PROGRESS CHECK ptitSeb/box86#249)
To do so, we'll use an awesome x86 emulator for ARM developed by @ptitSeb called box86.
But first, you need to enable System V IPC under General Setup in the kernel config and recompile it again. That's because the steam binary uses some semaphore functions and will crash in case it can't use them.
Next, we hit a problem: box86 can only be compiled by a 32 bit toolchain. But, in fact, this can be easily circumvented by using a 32 bit container:
$ sudo docker run -ti \
--net="host" \
--dns="8.8.8.8" \
-e DISPLAY=$DISPLAY \
-w /root \
-v $TMPDIR/.X11-unix:/tmp/.X11-unix \
-v $HOME/.Xauthority:/root/.Xauthority \
arm32v7/ubuntu
Note: if your system is 32 bit (run uname -m to check), you don't need to specify it with arm32v7/ubuntu. Simply using ubuntu instead will already use a 32 bit image.
Now that we are inside the container, let's install the tools we're gonna use, as well as the steam .deb installer:
# echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf
# apt update
# apt install wget libvdpau1 libappindicator1 libnm0 libdbusmenu-gtk4
# wget https://steamcdn-a.akamaihd.net/client/installer/steam.deb
# tar tf data.tar.xz \
| grep -v ".*\/$" \
| grep -o "[^\.].*\/" \
| uniq \
| while read folder; do
mkdir -p $folder;
done
wget http://mirrors.kernel.org/ubuntu/pool/universe/libv/libva/libva2_2.7.0-2_i386.deb wget LIBGL: Cannot use eglCopyBuffers, disabling it's use: LIBGL: ERROR: EGL Error detected: EGL_BAD_NATIVE_PIXMAP LIBGL: ERROR: EGL Error detected: EGL_BAD_CONTEXT
- kernel/Makefile
- net/netfilter/xt_qtaguid.c
--- orig/net/netfilter/xt_qtaguid.c 2020-05-12 12:13:14.000000000 +0300
+++ my/net/netfilter/xt_qtaguid.c 2019-09-15 23:56:45.000000000 +0300
@@ -737,7 +737,7 @@
{
struct proc_iface_stat_fmt_info *p = m->private;
struct iface_stat *iface_entry;
- struct rtnl_link_stats64 dev_stats, *stats;
+ struct rtnl_link_stats64 *stats;
struct rtnl_link_stats64 no_dev_stats = {0};
@@ -745,13 +745,8 @@
current->pid, current->tgid, from_kuid(&init_user_ns, current_fsuid()));
iface_entry = list_entry(v, struct iface_stat, list);
+ stats = &no_dev_stats;
- if (iface_entry->active) {
- stats = dev_get_stats(iface_entry->net_dev,
- &dev_stats);
- } else {
- stats = &no_dev_stats;
- }
/*
* If the meaning of the data changes, then update the fmtX
* string.- vendor/github.com/containerd/containerd/platforms/database.go
- scripts/docs/generate-man.sh
- man/md2man-all.sh
- cli/config/config.go
- runtime/v1/linux/bundle.go
- runtime/v2/shim/util_unix.go
- Makefile
- platforms/database.go
- vendor/github.com/cpuguy83/go-md2man/v2/md2man.go
I'd like to thank the Termux Dev team for this wonderful app and @xeffyr for discovering about the bug in net/netfilter/xt_qtaguid.c and sharing the patch, as well as all the conversation we had here that led to docker finally working.
If you are a docker developer reading this, please consider adding an official support for Android. Look above the possibilities it opens for a smartphone. If you are not a docker developer, consider supporting this by showing interest here. If we annoy the devs enough, this may become official (of they may simply unsubscribe from the thread and let it rot in the Issues section ¯\_(ツ)_/¯ ).