LXC for Android 9.0.0 r18 Goldfish

01_Android_LXC_Note.md

Steps to build Android kernel and system for LXC:

1. Download Android SDK, NDK, system and kernel source code;
2. Download modified lxc and libcap source code (ppoffiec/lxc-android, ppoffice/libcap-android)
3. Build Android system from source first so that we can link to its shared library when we build lxc (or maybe we don't have to);
4. Modify Android kernel config and make sure lxc-checkconfig does not report any missing features. Then build kernel;
5. Modify Android system/core/rootdir/init.rc according to init.rc.diff. Make sure you mount all the cgroup subsystems needed (cpu, cpuacct, cpuset, memory, pids, devices, freezer, blkio) under /sys/fs/cgruop, otherwise lxc won't find them;
6. Rebuilt the Android system image so that the modified init.rc is in the image;
7. Fire up the emulator with custom kernel (emulator -kernel kernel-file-path) and use adb shell and dmesg to check if everything is doing ok.

Steps to build and run LXC:

1. Build libcap and lxc following the instructions at https://github.com/clondroid/lxc-for-Android-7.1.2
2. adb push the built LXC and libcap binaries to the emulator, make sure the absolute path for the binaries is exactly the same as it is written in the build env file ($LXC_OUT_ROOT or $ODM);
3. Install termux for helpful third-party commands. Install tar from termux, which will be used by the lxc-download template;
4. adb root and adb shell into the emulator, set the following environment variables:

   export PATH=/data/data/com.termux/files/usr/bin:$PATH
   export LD_LIBRARY_PATH=/data/data/com.termux/files/usr/lib:$LD_LIBRARY_PATH
   export HOME=/data/data/com.termux/files/home
   

   /data/data/com.termux/files/usr/ here is the $LXC_OUT_ROOT you used previously. After this, we can use termux commands and tools from the adb shell.
5. Create an Ubuntu container from the adb shell: lxc-create -n ubuntu -t download -- --no-validate;
6. Edit the container configuration file in $LXC_OUT_ROOT/containers/ubuntu/config, delete the lxc.net.0.link line as we can't use bridge here to connect the container to the internet;
7. lxc-start -n ubuntu -F to start the container in the foreground. If everything is fine, you should see the login prompt although you can't login using user name and password.
8. Start a new adb shell with the same environment variables, run lxc-attach -n ubuntu -- /bin/bash to obtain a shell access to the contianer. Note that this shell does not have the environment variables set, and you cannot run commands by name.
9. In the bash, use /usr/bin/passwd root to set a password for the root user, and log into this user from the previously opened login prompt.

Setup the network for LXC container

1. In the adb shell, use lxc-info -n ubuntu to find the veth interface the container is connected to;
2. Give that veth interface an IP address such as 172.16.0.1/24 using ip addr add 172.16.0.1/24 dev interface_name;
3. Set up the routing table for the container subnet using ip route add default 172.16.0.0/24 via 172.16.0.1. It is very possible that the routing table with highest priority is not the default table. In this case, you need to check the routing rules using ip rules. If you decide to craete a new routing table for the container network, don't forget to update the ip rule (you may also need to delete some default rules);
4. Enable IP forward sysctl -w net.ipv4.ip_forward=1 and add an iptables NAT rule iptables -t nat -A POSTROUTING -o outgoing_interface -j MASQUERADE;
5. Go to the container shell, add an IP address to the default interface eth0 using ip addr add 172.16.0.2/24 dev eth0, a route using ip route add default via 172.16.0.1, and a DNS nameserver in /etc/resolv.conf. You can also make these changes permanent in /etc/network/interfaces;
6. Android poses restrictions on what users can access the network. So remember to add the root user to the inet and net_raw groups in /etc/group in the container:

   inet:x:3003:root
   net_raw:x:3004:root
   

7. You should be able to ping outside servers by now. However, apt will not work due to the Android network group issue mentioned in the last step. You can resolve this by removing the _apt user from /etc/passwd.

envsetup.sh

# Please see https://github.com/clondroid/lxc-for-Android-7.1.2

export ANDROID_SDK_HOME=/home/ppoffice/Android/Sdk
export ANDROID_NDK_HOME=/home/ppoffice/Android/android-ndk-r18b
export ANDROID_STANDALONE_TOOLCHAIN_HOME=/home/ppoffice/Android/lxc/toolchain
export SYSROOT=$ANDROID_STANDALONE_TOOLCHAIN_HOME/sysroot

# User specific environment and startup programs
PATH=${ANDROID_NDK_HOME}
PATH=$PATH:${ANDROID_SDK_HOME}/tools:${ANDROID_SDK_HOME}/platform-tools
PATH=$PATH:${ANDROID_STANDALONE_TOOLCHAIN_HOME}/bin:/usr/local/sbin:/usr/local/bin
PATH=$PATH:/usr/sbin:/usr/bin:/sbin:/bin
export PATH=$PATH

# Tell configure what tools to use.
export BUILD_TARGET_HOST=x86_64-linux-android
export AR=$BUILD_TARGET_HOST-ar
export AS=$BUILD_TARGET_HOST-clang
export CC=$BUILD_TARGET_HOST-clang
export CXX=$BUILD_TARGET_HOST-clang++
# export LD=$BUILD_TARGET_HOST-ld
export STRIP=$BUILD_TARGET_HOST-strip
export RANLIB=$BUILD_TARGET_HOST-ranlib

# Tell configure what flags Android requires.
export CFLAGS="-fPIE -fPIC --sysroot=$SYSROOT"
export LDFLAGS="-pie"

# SELinux specifics
BASEDIR=$(pwd)
export ANDROID_LIBS="/home/ppoffice/Android/lxc/include"
export CFLAGS="$CFLAGS -I$ANDROID_LIBS/include"
export LDFLAGS="$LDFLAGS -L$ANDROID_LIBS/lib"

export LXC_OUT_ROOT="/data/data/com.termux/files/usr/"

example_container_config.txt

# Template used to create this container: /data/data/com.termux/files/usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --no-validate
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /data/data/com.termux/files/usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /data/data/com.termux/files/usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.arch = linux64

# Container specific configuration
lxc.rootfs.path = dir:/data/data/com.termux/files/usr//containers/ubuntu/rootfs
lxc.uts.name = ubuntu

# Network configuration
lxc.net.0.type = veth
#lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:b8:16:74

init.rc.diff

diff --git a/rootdir/init.rc b/rootdir/init.rc
index b9464e7fd..e90c58f79 100644
--- a/rootdir/init.rc
+++ b/rootdir/init.rc
@@ -158,33 +158,25 @@ on init
 
     # sets up initial cpusets for ActivityManager
     mkdir /dev/cpuset
-    mount cpuset none /dev/cpuset nodev noexec nosuid
+    mount cpuset none /dev/cpuset nodev noexec nosuid cpuset
+    write /dev/cpuset/cgroup.clone_children 1
 
     # this ensures that the cpusets are present and usable, but the device's
     # init.rc must actually set the correct cpus
     mkdir /dev/cpuset/foreground
-    copy /dev/cpuset/cpus /dev/cpuset/foreground/cpus
-    copy /dev/cpuset/mems /dev/cpuset/foreground/mems
     mkdir /dev/cpuset/background
-    copy /dev/cpuset/cpus /dev/cpuset/background/cpus
-    copy /dev/cpuset/mems /dev/cpuset/background/mems
 
     # system-background is for system tasks that should only run on
     # little cores, not on bigs
     # to be used only by init, so don't change system-bg permissions
     mkdir /dev/cpuset/system-background
-    copy /dev/cpuset/cpus /dev/cpuset/system-background/cpus
-    copy /dev/cpuset/mems /dev/cpuset/system-background/mems
 
     # restricted is for system tasks that are being throttled
     # due to screen off.
     mkdir /dev/cpuset/restricted
-    copy /dev/cpuset/cpus /dev/cpuset/restricted/cpus
-    copy /dev/cpuset/mems /dev/cpuset/restricted/mems
 
     mkdir /dev/cpuset/top-app
-    copy /dev/cpuset/cpus /dev/cpuset/top-app/cpus
-    copy /dev/cpuset/mems /dev/cpuset/top-app/mems
+    mkdir /dev/cpuset/camera-daemon
 
     # change permissions for all cpusets we'll touch at runtime
     chown system system /dev/cpuset
@@ -199,6 +191,8 @@ on init
     chown system system /dev/cpuset/system-background/tasks
     chown system system /dev/cpuset/top-app/tasks
     chown system system /dev/cpuset/restricted/tasks
+    chown system system /dev/cpuset/camera-daemon
+    chown system system /dev/cpuset/camera-daemon/tasks
 
     # set system-background to 0775 so SurfaceFlinger can touch it
     chmod 0775 /dev/cpuset/system-background
@@ -209,7 +203,26 @@ on init
     chmod 0664 /dev/cpuset/top-app/tasks
     chmod 0664 /dev/cpuset/restricted/tasks
     chmod 0664 /dev/cpuset/tasks
-
+    chmod 0664 /dev/cpuset/camera-daemon/tasks
+
+    # mount cgroup subsystems in /sys/fs/cgroup
+    mount tmpfs cgroup_root /sys/fs/cgroup
+    mkdir /sys/fs/cgroup/cpu
+    mkdir /sys/fs/cgroup/cpuacct
+    mkdir /sys/fs/cgroup/cpuset
+    mkdir /sys/fs/cgroup/memory
+    mkdir /sys/fs/cgroup/pids
+    mkdir /sys/fs/cgroup/devices
+    mkdir /sys/fs/cgroup/freezer
+    mkdir /sys/fs/cgroup/blkio
+    mount cgroup none /sys/fs/cgroup/cpu nodev noexec nosuid cpu
+    mount cgroup none /sys/fs/cgroup/cpuacct nodev noexec nosuid cpuacct
+    mount cpuset none /sys/fs/cgroup/cpuset nodev noexec nosuid cpuset
+    mount cgroup none /sys/fs/cgroup/memory nodev noexec nosuid memory
+    mount cgroup none /sys/fs/cgroup/pids nodev noexec nosuid pids
+    mount cgroup none /sys/fs/cgroup/devices nodev noexec nosuid devices
+    mount cgroup none /sys/fs/cgroup/freezer nodev noexec nosuid freezer
+    mount cgroup none /sys/fs/cgroup/blkio nodev noexec nosuid blkio
 
     # qtaguid will limit access to specific data based on group memberships.
     #   net_bw_acct grants impersonation of socket owners.

libcap-android\build.sh

# Please see https://github.com/clondroid/lxc-for-Android-7.1.2

# libcap specifics
export DESTDIR=$ANDROID_LIBS
export lib=lib
export prefix=/

make -e

make install

lxc-android\build.sh

# Please see https://github.com/clondroid/lxc-for-Android-7.1.2

export PREFIX=$LXC_OUT_ROOT

make distclean

./autogen.sh

CPPFLAGS="$CPPFLAGS $CFLAGS" \
./configure --host=$BUILD_TARGET_HOST --enable-shared=yes --enable-static=yes \
	--disable-api-docs \
	--disable-selinux \
	--enable-capabilities \
	--disable-examples \
	--disable-lua \
	--disable-python \
	--disable-bash \
	--enable-configpath-log \
        \
        --disable-doc \
        --disable-api-docs \
	\
	--prefix=$PREFIX \
	--with-systemdsystemunitdir=$PREFIX/lib/systemd/system \
	--with-config-path=$LXC_OUT_ROOT/containers \
	--with-global-conf=$LXC_OUT_ROOT/.config \
	--with-runtime-path=/cache

make -j8
make install

x86_64_ranchu_defconfig.patch

77c77
< # CONFIG_FHANDLE is not set
---
> CONFIG_FHANDLE=y
152,154c152,155
< # CONFIG_CGROUP_PIDS is not set
< # CONFIG_CGROUP_DEVICE is not set
< # CONFIG_CPUSETS is not set
---
> CONFIG_CGROUP_PIDS=y
> CONFIG_CGROUP_DEVICE=y
> CONFIG_CPUSETS=y
> CONFIG_PROC_PID_CPUSET=y
156c157,161
< # CONFIG_MEMCG is not set
---
> CONFIG_PAGE_COUNTER=y
> CONFIG_MEMCG=y
> CONFIG_MEMCG_SWAP=y
> CONFIG_MEMCG_SWAP_ENABLED=y
> # CONFIG_MEMCG_KMEM is not set
158c163
< # CONFIG_CGROUP_PERF is not set
---
> CONFIG_CGROUP_PERF=y
163,164c168,171
< # CONFIG_BLK_CGROUP is not set
< # CONFIG_CHECKPOINT_RESTORE is not set
---
> CONFIG_BLK_CGROUP=y
> # CONFIG_DEBUG_BLK_CGROUP is not set
> CONFIG_CGROUP_WRITEBACK=y
> CONFIG_CHECKPOINT_RESTORE=y
168c175
< # CONFIG_USER_NS is not set
---
> CONFIG_USER_NS=y
315a323
> # CONFIG_BLK_DEV_THROTTLING is not set
348a357
> # CONFIG_CFQ_GROUP_IOSCHED is not set
479a489
> # CONFIG_MEM_SOFT_DIRTY is not set
722c732
< # CONFIG_PACKET_DIAG is not set
---
> CONFIG_PACKET_DIAG=y
724c734
< # CONFIG_UNIX_DIAG is not set
---
> CONFIG_UNIX_DIAG=y
805a816
> CONFIG_BRIDGE_NETFILTER=m
864c875
< # CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
---
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=y
923a935
> # CONFIG_NETFILTER_XT_MATCH_PHYSDEV is not set
988c1000,1001
< # CONFIG_NF_NAT_IPV6 is not set
---
> CONFIG_NF_NAT_IPV6=m
> CONFIG_NF_NAT_MASQUERADE_IPV6=m
1006c1019,1022
< # CONFIG_IP6_NF_NAT is not set
---
> CONFIG_IP6_NF_NAT=m
> CONFIG_IP6_NF_TARGET_MASQUERADE=m
> # CONFIG_IP6_NF_TARGET_NPT is not set
> # CONFIG_BRIDGE_NF_EBTABLES is not set
1013c1029,1032
< # CONFIG_BRIDGE is not set
---
> CONFIG_STP=m
> CONFIG_BRIDGE=m
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> # CONFIG_BRIDGE_VLAN_FILTERING is not set
1016c1035,1037
< # CONFIG_VLAN_8021Q is not set
---
> CONFIG_VLAN_8021Q=m
> # CONFIG_VLAN_8021Q_GVRP is not set
> # CONFIG_VLAN_8021Q_MVRP is not set
1017a1039
> CONFIG_LLC=m
1100c1122
< # CONFIG_NETLINK_DIAG is not set
---
> CONFIG_NETLINK_DIAG=y
1526c1548,1549
< # CONFIG_MACVLAN is not set
---
> CONFIG_MACVLAN=m
> # CONFIG_MACVTAP is not set
2004c2027
< # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
---
> CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
3481a3505
> CONFIG_EXPORTFS=y
3532c3556
< # CONFIG_PROC_CHILDREN is not set
---
> CONFIG_PROC_CHILDREN=y

This may be helpful