Skip to content

Instantly share code, notes, and snippets.

@arno01
Last active October 19, 2023 07:48
Show Gist options
  • Save arno01/ebf570af208e28c1a0cf78da4f63bc9c to your computer and use it in GitHub Desktop.
Save arno01/ebf570af208e28c1a0cf78da4f63bc9c to your computer and use it in GitHub Desktop.
Docker on Android

WORK IN PROGRESS

Docker on Android

Setup:

Samsung Galaxy Tab S5e SM-T720
Android Pie on Linux 4.9.112 (not rooted)
Termux
golang 1.12

Client

This will install the docker client to your ~/go/bin/ directory.

go get github.com/docker/cli/cmd/docker

The client is working, you can export DOCKER_HOST value to work with the dockerd, for example:

# export DOCKER_HOST=unix://$HOME/docker.sock
export DOCKER_HOST=tcp://192.168.X.Y:2376
docker run hello-world

Server

go get -u -d github.com/docker/docker/cmd/dockerd

rm -vf ~/go/src/github.com/docker/docker/daemon/graphdriver/register/register_btrfs.go
rm -vf ~/go/src/github.com/docker/docker/daemon/graphdriver/register/register_devicemapper.go

cd ~/go/src/github.com/docker/docker/cmd/dockerd
go install

containerd

containerd is the container runtime used by dockerd.

go get -u -d github.com/containerd/containerd/cmd/containerd
rm -vf ~/go/src/github.com/containerd/containerd/cmd/containerd/builtins_btrfs_linux.go
cd ~/go/src/github.com/containerd/containerd/cmd/containerd
go install

rootless docker

  1. Install rootlesskit
source ~/go/src/github.com/docker/docker/hack/dockerfile/install/rootlesskit.installer
REFIX=$GOPATH/bin _install_rootlesskit
  1. Install slirp4netns
git clone -b v0.3.0 https://github.com/rootless-containers/slirp4netns.git
cd slirp4netns
./autogen.sh
./configure --prefix=$PREFIX
make
make install
  1. Run rootless dockerd
~/go/src/github.com/docker/docker/contrib/dockerd-rootless.sh --experimental

Issue:

Apparently non-rooted Android is not permitting using the namespaces, probably due to SELinux rules or any other means such as unprivileged_userns_clone set to 0.. please try this if you have rooted Android.

+ exec rootlesskit --net=slirp4netns --mtu=65520 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run /data/data/com.termux/files/home/go/src/github.com/docker/docker/contrib/dockerd-rootless.sh --experimental
WARN[0000] "builtin" port driver is experimental
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
$ strace rootlesskit --net=slirp4netns --mtu=65520 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run bash |& grep CLONE_NEWUSER
clone(child_stack=NULL, flags=CLONE_NEWUSER|SIGCHLD) = -1 EPERM (Operation not permitted)

$ strace unshare -U id |& grep PERM
unshare(CLONE_NEWUSER)                  = -1 EPERM (Operation not permitted)

Refs

my go env

The defaults I have on my config:

$ go env
GOARCH="arm64"
GOBIN=""
GOCACHE="/data/data/com.termux/files/home/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="android"
GOOS="android"
GOPATH="/data/data/com.termux/files/home/go"
GOPROXY=""
GORACE=""
GOROOT="/data/data/com.termux/files/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/data/data/com.termux/files/usr/lib/go/pkg/tool/android_arm64"
GCCGO="gccgo"
CC="aarch64-linux-android-clang"
CXX="aarch64-linux-android-clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/data/data/com.termux/files/usr/tmp/go-build067260183=/tmp/go-build -gno-record-gcc-switches"
@FreddieOliveira
Copy link

FreddieOliveira commented Jun 14, 2020

For those who are interested, docker runs fine in android as long as you have an appropriated kernel (use https://github.com/moby/moby/blob/master/contrib/check-config.sh to check it). I'm running it right now, here's the prove:

This is a screenshot of docker running in my Redmi Note 7 device in termux. No chroot into a Linux rootfs and no qemu emulation of a Linux distro. This is docker in pure android.

But, there's a problem. When you pull a container, docker checks your device architecture and.operating system to download a container that matches it. To do so, it checks the container's manifest and see what architectures and OS's are supported by it and then pulls the correct one. Despite aarch64 is largely supported by most of the containers, the Android OS is not. This makes docker useless in Android, since there's no container that runs on it:

Screenshot_20200614-044554_Termux

For more info and alternatives on how to run docker on Android check this thread: termux/termux-root-packages#60

@nobhobbor
Copy link

if containers could run in android, then they could keep apps from calling home, which would defeat their purpose as far as google is concerned. i assume you know about the existence of the mobile open source OSs. if you want help or suggestions on how to proceed in android, Rob may be interested in what you have done here as he has done similar focused work. he offers de googled phones and is a developer. https://www.youtube.com/channel/UCYVU6rModlGxvJbszCclGGw but must have projects here in github too. it would be great to have an android kernel that supports running containers with google apps inside, but do they have complex code that prevents that?

@hessam94
Copy link

Could you RUN dockerd on your device. I tried it on a few real devices and android-x86. ALL of them says cgroup is not configured so now i need to rebuild the kernel to enable the cgroup which is almost hard. any suggestion, thanks.

@FreddieOliveira
Copy link

@nobhobbor Docker can't help with privacy and there's no point on installing it for this purpose. Here's why:

  1. Docker can only run containers that were created specifically for it, which means you can't run an Android apk inside docker. This defeats the idea of using docker for app isolation.

  2. Android apps are already isolated. Each app you install creates a new user and that user has only permissions within the folder the app was installed and permissions you explicitly give it. So, rule of thumb, don't give apps unnecessary permissions.

  3. Some permissions, like access to your phone's info (model, OS version, language, etc) and internet connection, are automatically granted and you can't change it. The best you can do is to use a firewall to completely block internet access for apps that actually don't need it. AFWall+ is a great one based on iptables, but requires a rooted device.

  4. If you still have to run a non trusted app which requires internet access and your phone's storage access to properly work (i.e. WhatsApp), you can further isolate it by using a work profile with apps like Island (actively maintaines, but not available in F-droid because it uses Google Firebase) or Shell (not actively maintained, but present in F-Droid).

Besides Rob's channel there's also The Hated One and others videos that contains good info about Android privacy.

If privacy is really a concern, what I recommend is to forget about docker, get rid of your phone's default OS and replace it with a trusted alternative (i.e. LineageOS) without gapps. Use open source apps alternatives to your old apps, like NewPipe instead of YouTube, OpenStreetMaps instead of Goggle Maps, etc. Follow the tips I gave above about permissions, firewall and work profile isolation. Use orbot 100% of time with VPN mode enabled to route every internet request through tor network. And mostly important, discipline yourself.

@hessam94 Yes, naturally dockerd, containerd and runc are successfully running too, otherwise the docker-cli wouldn't had correctly ran as showed. To run dockerd and the others daemons you have to compile your kernel with cgroups, namespaces and others features enabled. As I said, use https://github.com/moby/moby/blob/master/contrib/check-config.sh to check what still needs to be enabled in your kernel. When compiling it, run the menuconfig and type / to open a search window and enter the feature name without the CONFIG_. It will show you where the feature is located and its dependences.

After having an appropriated kernel you need to mount the cgroups hierarchy manually, see https://github.com/termux/termux-root-packages/blob/master/packages/lxc/lxc-setup-cgroups.sh as reference.

@nobhobbor
Copy link

nobhobbor commented Jun 14, 2020 via email

@hnmn
Copy link

hnmn commented Sep 30, 2020

@FreddieOliveira so , you successfully run docker in android
can make gist like this how you achieve docker in android.
may be step-by-step guide will be much appreciated .
because you are the one person done this.

@FreddieOliveira
Copy link

Hey @hnmn. Yes, good news on this, I was also able to run containers on it! 🎉 I even opened a feature request at the official docker repo, so they could add support for Android: moby/moby#41111. If you guys wanna see this become reality consider making some noise there, so the developers realize this is a really wanted feature.

About writing a tutorial on how to achieve this, bare in mind that's not a simple process. You'll have to compile your phone kernel, patch it, mess with it, download docker source code, patch it also, compile it, etc. But in any case, a step-by-step guide is something that's on my todo list. I'll post the link here when it's done.

@hessam94
Copy link

hessam94 commented Oct 1, 2020

I could run any container on andriod, but my problem is after the very first run the whole system crashes. so it means i am able to run container only one time and then android crashes. Any idea you guys have?
another strange thing is a cpuset directory is created on root folder on android which is different from linux. So i have to manually make a few folders and files in cpuset directory. it is hard to explain here, maybe you can guess something. thanks

@FreddieOliveira
Copy link

@hessam94 Your system crashes because of a bug in Android kernel. You'll have to patch it and then recompile it. This patch was mentioned here: termux/termux-root-packages#60 (comment)

The cpuset thing was also mentioned there. I highly recommend reading that thread, since all my steps and problems I faced are mentioned there.

My step-by-step guide will be basically that thread condensed.

@hessam94
Copy link

hessam94 commented Oct 1, 2020

@FreddieOliveira Thanks for your response, one point is I have already recompiled the android kernel to enable cgroup and some other features. should I patch it before or after recompilation? or it is not related ?

@FreddieOliveira
Copy link

The fetaures you enabled in the kernel and the bug that causes the crash are not related. There's no specific order on which you should do first. Just make sure you did both of them (enabling the features using make menuconfig and aplying the patch) before compiling the kernel.

@hnmn
Copy link

hnmn commented Oct 1, 2020

@FreddieOliveira thanks again for quick reply.
i have compile kernel but problem i don't khow where to modify source code (i.e patch) . as you mentioned we need to modify n complile various
packages which are discussed in various issues .i just want to make gist or repo where all the pieces are present that would be easy .

@mad-ady
Copy link

mad-ady commented Dec 29, 2020

Hey guys!
I'm trying to get Docker running in a Linux chroot (Ubuntu 18 via Linux Deploy) running on Lineage 17 on a Odroid N2 (with a kernel compiled with docker support: http://paste.ubuntu.com/p/TbXrX4yWHN/).
I followed your steps, except I ran them in the Linux chroot, not in termux.
I am able to start containerd:

# ~/go/bin/containerd
INFO[2020-12-29T13:23:06.634962961+02:00] starting containerd                           revision= version=1.4.0+unknown
INFO[2020-12-29T13:23:06.714363003+02:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2020-12-29T13:23:06.714504920+02:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
WARN[2020-12-29T13:23:06.714572878+02:00] failed to load plugin io.containerd.snapshotter.v1.aufs  error="invalid aufs configuration"
INFO[2020-12-29T13:23:06.714609753+02:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2020-12-29T13:23:06.714656253+02:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2020-12-29T13:23:06.714688420+02:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2020-12-29T13:23:06.714752128+02:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2020-12-29T13:23:06.714978170+02:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
WARN[2020-12-29T13:23:06.715087586+02:00] failed to load plugin io.containerd.snapshotter.v1.zfs  error="invalid zfs configuration"
INFO[2020-12-29T13:23:06.715121795+02:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2020-12-29T13:23:06.715164420+02:00] could not use snapshotter zfs in metadata plugin  error="invalid zfs configuration"
WARN[2020-12-29T13:23:06.715201336+02:00] could not use snapshotter aufs in metadata plugin  error="invalid aufs configuration"
WARN[2020-12-29T13:23:06.715227461+02:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2020-12-29T13:23:06.715257628+02:00] metadata content store policy set             policy=shared
INFO[2020-12-29T13:23:06.718472336+02:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2020-12-29T13:23:06.718630503+02:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2020-12-29T13:23:06.718803128+02:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.718951336+02:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719033836+02:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719083003+02:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719135628+02:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719185961+02:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719237670+02:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719289211+02:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.719367170+02:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2020-12-29T13:23:06.721884711+02:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2020-12-29T13:23:06.724095461+02:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2020-12-29T13:23:06.729475295+02:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2020-12-29T13:23:06.729670045+02:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2020-12-29T13:23:06.730064461+02:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730214878+02:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730340920+02:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730444878+02:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730535711+02:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730626003+02:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730715086+02:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730793670+02:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.730864170+02:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2020-12-29T13:23:06.731665753+02:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.732201920+02:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.732691336+02:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.733175170+02:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2020-12-29T13:23:06.735870420+02:00] serving...                                    address=/run/containerd/containerd.sock.ttrpc
INFO[2020-12-29T13:23:06.736949170+02:00] serving...                                    address=/run/containerd/containerd.sock
INFO[2020-12-29T13:23:06.737416045+02:00] containerd successfully booted in 0.112886s

... and dockerd ...

# ~/go/bin/dockerd --iptables=false
INFO[2020-12-29T13:24:40.216266256+02:00] Starting up
INFO[2020-12-29T13:24:40.220255673+02:00] parsed scheme: "unix"                         module=grpc
INFO[2020-12-29T13:24:40.220327714+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-12-29T13:24:40.220379298+02:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2020-12-29T13:24:40.220406673+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-12-29T13:24:40.225100923+02:00] parsed scheme: "unix"                         module=grpc
INFO[2020-12-29T13:24:40.225250214+02:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-12-29T13:24:40.225378256+02:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2020-12-29T13:24:40.225419923+02:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-12-29T13:24:40.260867881+02:00] [graphdriver] using prior storage driver: overlay2
WARN[2020-12-29T13:24:40.272595506+02:00] Your kernel does not support CPU CFS scheduler
WARN[2020-12-29T13:24:40.272657589+02:00] Unable to find blkio cgroup in mounts
INFO[2020-12-29T13:24:40.272974631+02:00] Loading containers: start.
INFO[2020-12-29T13:24:40.511562756+02:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2020-12-29T13:24:40.552736881+02:00] Loading containers: done.
INFO[2020-12-29T13:24:40.698796464+02:00] Docker daemon                                 commit=library-import graphdriver(s)=overlay2 version=library-import
INFO[2020-12-29T13:24:40.699200131+02:00] Daemon has completed initialization
INFO[2020-12-29T13:24:40.764358423+02:00] API listen on /var/run/docker.sock

ERRO[2020-12-29T13:30:53.216723267+02:00] stream copy error: reading from a closed fifo
ERRO[2020-12-29T13:30:53.216799475+02:00] stream copy error: reading from a closed fifo
ERRO[2020-12-29T13:30:53.378941059+02:00] a253f1025602c94c70f57e9e674d3d2a939ee43e9c5cbed828e9c8af979a5f63 cleanup: failed to delete container from containerd: no such container
ERRO[2020-12-29T13:30:53.379082517+02:00] Handler for POST /v1.41/containers/a253f1025602c94c70f57e9e674d3d2a939ee43e9c5cbed828e9c8af979a5f63/start returned error: runtime "io.containerd.runc.v2" binary not installed "containerd-shim-runc-v2": file does not exist: unknown

The errors reported by it are generated when I try to run docker run hello-world

# ~/go/bin/docker run hello-world
docker: Error response from daemon: runtime "io.containerd.runc.v2" binary not installed "containerd-shim-runc-v2": file does not exist: unknown.
ERRO[0000] error waiting for container: context canceled

Most likely I still need to start some components manually. Any idea what I'm missing?

@FreddieOliveira
Copy link

@mad-ady I don't think you need to start containerd yourself, just starting dockerd should be enough. The docker daemon starts its dependencies by itself. About your error it looks you don't have runc installed. You can apt install runc or compile it from source. Also, running dockerd with --debug flag gives more verbosity.

@mad-ady
Copy link

mad-ady commented Dec 31, 2020 via email

@FreddieOliveira
Copy link

@mad-ady I'm finishing writing the step by step guide I promised a while ago and ran into the same issue as you. I realized it's not the binary runc that's missing, it's actually a binary called containerd-shim-runc-v2. It's part of containerd and strangely enough this binary is not compiled by default when running make, you have to explicitly compile it with make bin/containerd-shim-runc-v2.

@hessam94
Copy link

hessam94 commented Jan 8, 2021

@FreddieOliveira I coudnt find time to run your patch , i will do it in 2 weeks thanks

@FreddieOliveira
Copy link

Don't worry, I intend to post the tutorial by the end of this week. You can use it as a reference.

@FreddieOliveira
Copy link

The step by step tutorial is finally done. Whoever is interested in get docker running on its own phone can check here for instructions.

@mad-ady
Copy link

mad-ady commented Jan 12, 2021

Thank you for your work. Hope to give it a try soon and provide feedback!

@hessam94
Copy link

hessam94 commented Feb 9, 2021

I am recompiling the android on a ubuntu Linux, would you help us how to apply the patch (segmentation fault patch) on android kernel before compilation a new kernel? I got the entire android files on linux and trying to build a new one
thanks

@xy815661276
Copy link

xy815661276 commented May 13, 2021

We open the Android container, which can run the docker container on the Android operating system. Not only can run Docker containers, but also other containers, such as podman. In addition, we have also enabled the container to support checkpoint and restore functions, and support the migration of containers across architectures and operating systems. See https://github.com/CGCL-codes/Android-Container for details.

@jelllyam
Copy link

jelllyam commented Jun 3, 2022

For those who are interested, docker runs fine in android as long as you have an appropriated kernel (use https://github.com/moby/moby/blob/master/contrib/check-config.sh to check it). I'm running it right now, here's the prove:

This is a screenshot of docker running in my Redmi Note 7 device in termux. No chroot into a Linux rootfs and no qemu emulation of a Linux distro. This is docker in pure android.

But, there's a problem. When you pull a container, docker checks your device architecture and.operating system to download a container that matches it. To do so, it checks the container's manifest and see what architectures and OS's are supported by it and then pulls the correct one. Despite aarch64 is largely supported by most of the containers, the Android OS is not. This makes docker useless in Android, since there's no container that runs on it:

Screenshot_20200614-044554_Termux

For more info and alternatives on how to run docker on Android check this thread: termux/termux-root-packages#60

Can you share your kernel in redmi note 7

@Sokan2004
Copy link

Verify Github on Galaxy. gid:wzXiXzmeFjYF4QKxtCUfnP

@FreddieOliveira
Copy link

@jelllyam, here's the kernel I'm currently using. It's based on LineageOS 17.1 (Android 10): https://ufile.io/a6guqaku

And out of curiosity, here's my kernel config file used when compiling it: https://ufile.io/5u3q10qu

The download links will be valid for 30 days.

Be aware that this kernel is for Android 10. Flashing it on a different version may cause boot loop. In case this happens, just reflash your ROM. You won't lose any of your data flashing the kernel or your ROM.

To learn how to install and use docker after you successfully flash the kernel see https://gist.github.com/FreddieOliveira/efe850df7ff3951cb62d74bd770dce27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment