Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
alpine on cubietruck notes
# Host installation and setup
## Alpine binaries
Fetch and validate:
curl -LO http://dl-cdn.alpinelinux.org/alpine/v3.4/releases/armhf/alpine-uboot-3.4.0-armhf.tar.gz
curl -LO http://dl-cdn.alpinelinux.org/alpine/v3.4/releases/armhf/alpine-uboot-3.4.0-armhf.tar.gz.asc
gpg -v alpine-uboot-3.4.0-armhf.tar.gz.asc
Contents:
alpine-uboot-3.4.0-armhf
├── apks
│   └── armhf
│   ├── alpine-base-3.4.0-r0.apk
[...]
│   └── zlib-1.2.8-r2.apk
├── boot
│   ├── dtbs
│   │   ├── am335x-baltos-ir5221.dtb
[...]
│   │   ├── sun7i-a20-cubieboard2.dtb
[...]
│   │   └── vexpress-v2p-ca9.dtb
│   ├── initramfs-grsec
│   ├── modloop-grsec
│   └── vmlinuz-grsec
├── extlinux
│   └── extlinux.conf
└── u-boot
├── am335x_boneblack
│   ├── MLO
[...]
├── Cubieboard2
│   └── u-boot.img
[...]
└── wandboard_solo
└── u-boot.imx
14 directories, 346 files
However we are going to ignore the `u-boot` and `kernel` (+`dtb`)
binaries included here since they don't boot (and for the kernel we
want to enable Xen and KVM support).
## Bootloader and Kernel
### Cross compilation environment
To compile the kernel and u-boot we need a suitable ARM cross
compiler. One is available in Debian Stretch:
apt install gcc-arm-linux-gnueabi
TODO: Dockerise builds using this instead
### Bootloader (u-boot)
Clone and build a `u-boot` binary:
git clone http://git.denx.de/u-boot.git
cd u-boot
git checkout v2016.05
make CROSS_COMPILE=arm-linux-gnueabi- Cubieboard2_defconfig
make CROSS_COMPILE=arm-linux-gnueabi- -j12
(if not using Debian cross compiler environment substitute CROSS_COMPILE to suite)
TODO: Dockerise build
### Kernel (linux)
Clone and build a Linux kernel binary:
git clone http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
cd linux
git remote add -f stable https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
git checkout v4.4.14 # Current latest LTS, stable/linux-4.4.y might also be suitable
make CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm sunxi_defconfig
# Tailor with Xen+KVM (needs LPAE), bridging, TUN/TAP, loop device, LVM, syncookies
cat >> .config <<EOF
CONFIG_ARCH_VIRT=y
CONFIG_ARM_LPAE=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_XEN=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_XEN_WDT=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_ARM_HOST=y
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI_VIRTIO=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VHOST=y
CONFIG_VHOST_NET=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_BRIDGE=y
CONFIG_TUN=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_UEVENT=y
CONFIG_SYN_COOKIES=y
EOF
make CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm olddefconfig
make CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm all -j12
Kernel binary is `arch/arm/boot/zImage`. `dtb` is `arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb`.
TBD: Dockerise build
## Setup SD card
On a host machine, identify device corresponding to the SD card
(`/dev/sda` here, YMMV). Take care since this process will **DESTROY**
anything on the device.
Repartition device with:
sudo fdisk /dev/sda
Then partition (short key presses listed):
* Create a new DOS parition table:
* `o`
* Create a new 128M FAT partition (type 6) as first partition:
* `n`
* `p`
* `1`
* _Enter_
* `+128M`
* `t`
* `6`
* Fill the rest of the device with an LVM (type e8) partition
* `n`
* `p`
* `2`
* _Enter_
* _Enter_
* `t`
* `2`
* `8e`
* Write and exit
* `w`
Result should be like this (but LVM partition size depends on your device):
sudo fdisk -l /dev/sda
Disk /dev/sda: 29.2 GiB, 31322013696 bytes, 61175808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x76667587
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 264191 262144 128M 6 FAT16
/dev/sda2 264192 61175807 60911616 29G 8e Linux LVM
Format the new VFAT partition:
sudo mkfs.vfat /dev/sda1
Nuke any lingering data on new LVM partition:
sudo dd if=/dev/zero bs=1M of=/dev/sda2 count=1
Mount the new FAT partition, extract the alpine tarball into it,
add our kernel + initrd and customized extlinux config:
sudo mount /dev/sda1 /mnt
sudo tar -C /mnt -xaf alpine-uboot-3.4.0-armhf.tar.gz
sudo cp linux/arch/arm/boot/zImage /mnt/boot/vmlinuz
sudo cp linux/arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb /mnt/boot
sudo dd of=/mnt/extlinux/extlinux.conf <<EOF
LABEL custom
MENU LABEL Custom kernel
LINUX /boot/vmlinuz
INITRD /boot/initramfs-grsec
DEVICETREEDIR /boot
APPEND modules=loop,squashfs,sd-mod,usb-storage console=\${console}
EOF
sudo umount /mnt
Install u-boot into boot sector:
sudo dd if=u-boot/u-boot-sunxi-with-spl.bin of=/dev/sda bs=1024 seek=8
# Boot and Setup Alpine Linux
## First Boot and Initial Setup
Choose a new MAC address for the board. You can use [randmac](https://www.hellion.org.uk/cgi-bin/randmac.pl) to generate a valid local MAC address.
I am setting up two systems and used:
* cubie-kvm: `fe:15:ec:93:03:3c`
* cubie-xen: `96:0f:f0:98:4c:44`
(do not use these if you are on the same VLAN as these systems!)
Insert SD card into the Cubieboard2, attach serial dongle and
boot. Press any key to interrupt boot and (re)initialise environment
(use your own MAC address from above!) with:
=> env default -f -a
## Resetting to default environment
=> setenv ethaddr "xx:xx:xx:xx:xx:xx"
e.g.
=> setenv ethaddr "fe:15:ec:93:03:3c"
=> setenv ethaddr "96:0f:f0:98:4c:44"
=> setenv localcmd run scan_dev_for_scripts
=> saveenv
Saving Environment to MMC...
Writing to MMC(0)... done
=>
(the `localcmd` is used when booting Xen below)
Reboot into alpine:
=> reset
Alpine will boot, login (root, no passwd). Run `setup-alpine`. Setup
desired hostname etc, the defaults for most things are fine. Store
config in `mmcblk0p1` (default, this is the same FAT partition as
created above).
## mdns
Install avahi:
apk update
apk add avahi dbus
echo net.ipv4.igmp_max_memberships=20 > /etc/sysctl.conf
service sysctl restart
service dbus start
service avahi-daemon start
rc-update add dbus
rc-update add avahi-daemon
lbu ci
At this point ssh to `$HOSTNAME.local` should work, so can switch from
serial to ssh if desired. To configure (persistent) ssh keys for root:
mkdir /root/.ssh
cat >>/root/.ssh/authorized_keys <<EOF
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9Fq8pzTh0ClNZQwZKOW5QT+YuO06fPbCEQebFR6iDSpy6mCg4r+StT43Ry8f2DobgXdzXWdY/Uf6CuC2aiv+XBFy1gjXkoZcquoB0tz9odlGZo+hBUSl+Bw2T3QuhgNuvyZmhH+GFoufSUKjtNZ2ROicme9k0v/CsGjW6jDo2Xl6ZoZhfcseA8loU+slj15JKckzcqKaTnNf9VXd/ATeVTV51L9tgTyeC8d7rLiAzk7LJlPzJPhq+K8r7fLMjoteXw1et3OUmKWzLWML6pFPxdp2Hr99VxlRf4m5vUKptPeyqMG9OfbWo+WWDZ9xKbV2LiDjnneoDP4it/ohqiUS7 ijc@bokrug
EOF
lbu include /root/.ssh/authorized_keys
lbu ci
## Upgrading
apk upgrade
lbu ci
## Configure LVM
Configure LVM:
apk add lvm2
service lvm start
rc-update add lvm
pvcreate /dev/mmcblk0p2
VG=`hostname`
vgcreate $VG /dev/mmcblk0p2
lbu ci
## Create guest block (LV) device:
lvcreate -n alpine-disk -L 4G /dev/$VG
fdisk /dev/$VG/alpine-disk
# Create a 128M vfat partition and the rest a Linux partition
apk add multipath-tools # for kpartx
kpartx -a /dev/$VG/alpine-disk
dmsetup mknodes # Should be automatic, not sure why not
mkfs.vfat /dev/mapper/${VG//-/--}-alpine--disk1
mount /dev/mapper/${VG//-/--}-alpine--disk1 /mnt/
cp -r /media/mmcblk0p1/apks /mnt/apks
umount /mnt
kpartx -d /dev/$VG/alpine-disk
dmsetup mknodes # Should be automatic, not sure why not
## Setup bridged networking
For running guests we need to setup a network bridge following
[https://wiki.alpinelinux.org/wiki/Bridge]()
apk add bridge
HN=`hostname`
cat >/etc/network/interfaces <<EOF
auto lo
iface lo inet loopback
auto br0
iface br0 inet dhcp
hostname $HN
bridge-ports eth0
bridge-stp 0
#auto eth0
iface eth0 inet dhcp
hostname $HN
EOF
lbu ci
# KVM Specific setup
## Host Configuration
This section assumes the hostname (and therefore the LVM VG created above) is "cubie-kvm".
apk add qemu-system-arm
cat >/etc/qemu-ifup <<"EOF"
#!/bin/sh
set -x
switch=br0
if [ -n "$1" ];then
ip link set $1 up
sleep 0.5s
brctl addif $switch $1
exit 0
else
echo "Error: no interface specified"
exit 1
fi
EOF
chmod +x /etc/qemu-ifup
lbu ci
`qemu-ifup` adjusted [http://www.linux-kvm.org/page/Networking]()
## Running an Alpine Guest
Boot e.g. the original Alpine binaries (not our local builds):
qemu-system-arm \
-nographic \
-enable-kvm -machine virt -cpu host \
-kernel /media/mmcblk0p1/boot/vmlinuz-grsec \
-initrd /media/mmcblk0p1/boot/initramfs-grsec \
--append 'console=ttyAMA0'
Which works but lacks disk and network. For virtio support we need to
use our kernel (same one as for the host).
qemu-system-arm \
-nographic \
-enable-kvm -machine virt -cpu host \
-kernel /media/mmcblk0p1/boot/vmlinuz \
-initrd /media/mmcblk0p1/boot/initramfs-grsec \
--append 'console=ttyAMA0 earlycon earlyprintk' \
-netdev tap,id=net0 -device virtio-net-device,mac=b2:82:6c:ef:9c:58,netdev=net0 \
-drive if=none,format=raw,file=/dev/mapper/cubie--kvm-alpine--disk,cache=writeback,id=blk -device virtio-blk-device,drive=blk
Configure as usual via `setup-alpine`.
To avoid needing to setup bridging we can use the (slower) userspace network stack:
qemu-system-arm \
-nographic \
-enable-kvm -machine virt -cpu host \
-kernel /media/mmcblk0p1/boot/vmlinuz \
-initrd /media/mmcblk0p1/boot/initramfs-grsec \
--append 'console=ttyAMA0 earlycon earlyprintk' \
-netdev user,id=net0 -device virtio-net-device,mac=b2:82:6c:ef:9c:58,netdev=net0 \
-drive if=none,format=raw,file=/dev/mapper/cubie--kvm-alpine--disk,cache=writeback,id=blk -device virtio-blk-device,drive=blk
This means we don't need to reconfigure `/etc/network/interfaces` with
`br0` above and do not need the `/etc/qemu-ifup` script.
## Running a Debian Guest
NB Jessie doesn't seem to boot. We use Stretch instead.
lvcreate -L8G -n debian /dev/cubie-kvm
cd /
wget ftp://ftp.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
wget ftp://ftp.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
qemu-system-arm \
-nographic \
-m 256 \
-enable-kvm -machine virt -cpu host \
-kernel /tmp/vmlinuz \
-initrd /tmp/initrd.gz \
--append 'console=ttyAMA0 debug' \
-netdev tap,id=net0 -device virtio-net-device,mac=d6:76:5d:44:fc:93,netdev=net0 \
-drive if=none,format=raw,file=/dev/mapper/cubie--kvm-debian,cache=writeback,id=blk -device virtio-blk-device,drive=blk
-netdev user,id=net0 -device virtio-net-device,mac=b2:82:6c:ef:9c:58,netdev=net0 \
Run through the install.
TBD: How to boot the resulting system. Need firmware?
# Xen Specific setup
## Host Configuration
This section assumes the hostname (and therefore the LVM VG created above) is "cubie-xen".
### Install Xen APKs
At the moment (Alpine 3.4) this requires enabling the Edge repository:
sed -i -e 's,^#\(.*/edge/main\)$,\1,g' /etc/apk/repositories
apk update
apk add xen xen-hypervisor
lbu ci
### Configure Xen boot
Add Xen to `/boot`
mount -o remount,rw /media/mmcblk0p1/
cp /boot/xen /media/mmcblk0p1/boot/xen
cat >/media/mmcblk0p1/extlinux/extlinux.conf <<EOF
LABEL local
MENU LABEL Local boot
LOCALBOOT 0
LABEL custom
MENU LABEL Custom kernel
LINUX /boot/vmlinuz
INITRD /boot/initramfs-grsec
DEVICETREEDIR /boot
APPEND modules=loop,squashfs,sd-mod,usb-storage console=\${console}
EOF
mount -o remount,ro /media/mmcblk0p1/
We need to select a suitable address to load Xen. The default u-boot environment has:
kernel_addr_r =0x42000000 16M @ 1056M
fdt_addr_r =0x43000000 3M @ 1072M
ramdisk_addr_r=0x43300000 ... @ 1075M
So steal the last 2M of 16M assigned for the kernel (our actual kernel is <4M) and use:
kernel_addr_r =0x42000000 14M @ 1056M
xen_addr_r =0x42e00000 2M @ 1070M
fdt_addr_r =0x43000000 3M @ 1072M
ramdisk_addr_r=0x43300000 ... @ 1075M
U-boot `mkimage` command is not currently in Alpine, so using
e.g. Debian and the `u-boot-tools` package create a boot script:
cat > boot.uscr <<"EOF"
setenv xen_addr_r 0x42e00000
load mmc 0 ${xen_addr_r} /boot/xen
load mmc 0 ${fdt_addr_r} /boot/${fdtfile}
load mmc 0 ${kernel_addr_r} /boot/vmlinuz
fdt addr ${fdt_addr_r}
fdt resize
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "modules=loop,squashfs,sd-mod,usb-storage console=hvc0 clk_ignore_unused rootflags=size=128M"
fdt set /chosen xen,xen-bootargs "conswitch=x dom0_mem=256M"
load mmc 0 ${ramdisk_addr_r} /boot/initramfs-grsec
bootz ${xen_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}
EOF
apt install u-boot-tools
mkimage -T script -A arm -d boot.uscr boot.scr
Copy the `boot.scr` to `/media/mmcblk0p1/boot.scr` on the cubietruck:
scp boot.scr root@cubie-xen.local:/media/mmcblk0p1/boot.scr
Reboot back into Alpine, but now under Xen. Configure dom0 with:
setup-xen-dom0
lbu ci
## Running an Alpine Guest
Boot e.g. our local built kernel with alpine initramfs
(vmlinuz-grsec does not work under Xen):
cat > /etc/xen/alpine.cfg <<EOF
name = "alpine"
kernel = "/media/mmcblk0p1/boot/vmlinuz"
ramdisk = "/media/mmcblk0p1/boot/initramfs-grsec"
cmdline = "console=hvc0"
memory = 128
vif = ["mac=06:ac:b4:92:fc:49,bridge=br0"]
disk = ["vdev=xvda,format=raw,target=/dev/cubie-xen/alpine-disk"]
EOF
lbu ci
xl cr -c /etc/xen/alpine.cfg
Configure as usual via `setup-alpine`
## Running a Debian Guest
NB Jessie doesn't seem to boot. We use Stretch instead.
lvcreate -L8G -n debian /dev/cubie-xen
cd /
wget ftp://ftp.debian.org/debian/dists/stretch/main/installer-armhf/current/images/netboot/vmlinuz
wget ftp://ftp.debian.org/debian/dists/stretch/main/installer-armhf/current/images/netboot/initrd.gz
cat > /etc/xen/debian.cfg <<EOF
name = "debian"
memory = 256
vcpus = 2
disk = ['phy:/dev/cubie-xen/debian,xvda,w']
vif = ['mac=02:ff:e4:2a:dc:1b,bridge=br0']
kernel = "/tmp/vmlinuz"
ramdisk = "/tmp/initrd.gz"
cmdline = "debian-installer/exit/always_halt=true -- console=hvc0"
EOF
lbu ci
xl cr -c /etc/xen/debian.cfg
Run through the install, ignoring the error about lack of bootloader.
At the end the system will halt, exit the guest console and destroy
the guest (shutdown appears to be broken):
xl dest debian
Rewrite the configuration file to allow booting the installed system
in a hacky way to allow fixing the bootloader:
cat > /etc/xen/debian.cfg <<EOF
name = "debian"
memory = 256
vcpus = 2
disk = ['phy:/dev/cubie-xen/debian,xvda,w']
vif = ['mac=02:ff:e4:2a:dc:1b,bridge=br0']
bootloader = "pygrub"
bootloader_args = ["--kernel=/vmlinuz", "--ramdisk=/initrd.img", "--args=console=hvc0 root=/dev/xvda2"]
EOF
lbu ci
xl cr -c /etc/xen/debian.cfg
Login to the guest console and:
apt install pv-grub-menu
You should now have a `/boot/grub/menu.lst`. However if your guest
`/boot` is separate from `/` (which is the default in guided
partitioning) then this will be wrong, due to [Debian bug
#771949](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771949). Fix it up based on the suggestion in [Message #25](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771949#25):
sed -i.bak -e 's,^kernel_dir=/boot,if [ "${root_device}" == "${boot_device}" ] ; then kernel_dir=/boot ; else kernel_dir= ; fi,' /usr/sbin/update-menu-lst
update-menu-lst
(an alternative would perhaps have been `ln -s /boot /boot/boot`)
Shutdown the system again and rewrite the config for the final time:
cat > /etc/xen/debian.cfg <<EOF
name = "debian"
memory = 256
vcpus = 2
disk = ['phy:/dev/cubie-xen/debian,xvda,w']
vif = ['mac=02:ff:e4:2a:dc:1b,bridge=br0']
bootloader = "pygrub"
EOF
lbu ci
xl cr -c /etc/xen/debian.cfg
Consider installing avahi-daemon
apt install avahi-daemon
For unikernels work:
apt install opam
To boot the host kernel:
cat > /etc/xen/debian.cfg <<EOF
name = "debian"
memory = 256
vcpus = 2
disk = ['phy:/dev/cubie-xen/debian,xvda,w']
vif = ['mac=02:ff:e4:2a:dc:1b,bridge=br0']
kernel = "/media/mmcblk0p1/boot/vmlinuz"
cmdline = "console=hvc0 root=/dev/xvda2"
EOF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment