Instantly share code, notes, and snippets.

Embed
What would you like to do?
Running Ubuntu 16.04.1 armhf on Qemu

Running Ubuntu 16.04.1 armhf on Qemu

This is a writeup about how to install Ubuntu 16.04.1 Xenial Xerus for the 32-bit hard-float ARMv7 (armhf) architecture on a Qemu VM via Ubuntu netboot.

The setup will create a Ubuntu VM with LPAE extensions (generic-lpae) enabled. However, this writeup should also work for non-LPAE (generic) kernels.

The performance of the resulting VM is quite good, and it allows VMs with >1G ram (compared to 256M on versatilepb and 1G on versatile-a9/versatile-a15). It also supports virtio disks whereas versatile-a9/versatile-a15 only support SD cards via the -sd argument.

Get netboot files

The netboot files are available on the official Ubuntu mirror. The following commands will download the kernel (vmlinuz) and initrd (initrd.gz) in a new directory called netboot:

mkdir netboot
cd netboot
wget -r -nH -nd -np -R "index.html*" --quiet http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/

Create a image file

The following command creates a image file that will be used as a disk for the Ubuntu system:

qemu-img create -f qcow2 ubuntu.img 16G

Setup networking

Installing Ubuntu via netboot requires an Internet connection. Therefore the Ubuntu VM requires a network intefaces that can use the Internet connection of the host system. The host systems needs to create a tuntap device for the VM and a bridge that will connect the tuntap interface with the Internet-facing interface. The bridge is called br0 and the tuntap device for the Ubuntu VM will be tap0.

sudo ip tuntap add dev tap0 mode tap
sudo ip link set up dev tap0
sudo ip link set tap0 master br0

Creating a bridge and adding the Internet-facing interface (e.g. eth0):

sudo ip link add br0 type bridge
sudo ip link set eth0 master br0

Allow IP packet forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

Start a DHCP server in the bridge so that the Ubuntu VM receives an IP address:

echo "subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.10 192.168.0.100;
  option routers 192.168.0.1;
  option domain-name-servers 208.67.222.222, 208.67.220.220;
}" > qemu-dhcpd.conf
sudo dhcpd -cf qemu-dhcpd.conf br0

Set router IP on bridge interface:

ip addr add 192.168.0.1/24 dev br0
ip link set up dev br0

Start the netboot installation

From the netboot directory (containing the netboot files vmlinuz and initrd.gz), run the following command:

qemu-system-arm \
  -kernel vmlinuz \
  -initrd initrd.gz \
  -append "root=/dev/ram" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -hda ubuntu.img

The kernel should now boot into the Ubuntu installer within the terminal window where the command has been executed.

Extract the new kernel

After the installation process has been finished, extract the kernel and initrd files from the new installed Ubuntu system. Mount the Ubuntu image:

qemu-img convert -f qcow2 -O raw ubuntu.img ubuntu-raw.img
sudo losetup /dev/loop0 ubuntu-raw.img
OFFSET=$(($(sudo fdisk -l /dev/loop0 |grep /dev/loop0p1 |awk '{print $3}')*512))
sudo mount -o loop,offset=$OFFSET /dev/loop0 /mnt

Create a new directory boot for the new kernel and initrd files and copy them into it:

mkdir boot
cp /mnt/initrd.img-4.4.0-38-generic-lpae
cp /mnt/vmlinuz-4.4.0-38-generic-lpae boot

Cleanup:

sudo umount /mnt
sudo losetup -d /dev/loop0
rm ubuntu-raw.img

TODO: Add working solution that does not require to convert the image file.

Start the Ubuntu VM

The Ubuntu VM directory should now have the following structure:

├── boot
│   ├── initrd.img-4.4.0-38-generic-lpae
│   └── vmlinuz-4.4.0-38-generic-lpae
├── netboot
│   ├── initrd.gz
│   └── vmlinuz
└── ubuntu.img

Start Qemu:

qemu-system-arm \
  -kernel boot/vmlinuz-4.4.0-38-generic-lpae \
  -initrd boot/initrd.img-4.4.0-38-generic-lpae \
  -append "root=/dev/vda2 rootfstype=ext4" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -monitor telnet:127.0.0.1:9000,server,nowait \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -drive file=ubuntu.img,if=virtio

The following script can be used for starting the VM:

#!/bin/sh
NET_IF=tap0
BRIDGE=br0
if ! grep --quiet $NET_IF /proc/net/dev;then
    echo "Creating ${NET_IF} device"
    sudo ip tuntap add dev $NET_IF mode tap
    sudo ip l s up dev $NET_IF
fi
if grep --quiet $BRIDGE /proc/net/dev;then
    sudo ip l s $NET_IF master $BRIDGE
else
    echo "${BRIDGE} does not exist!"
    exit
fi
qemu-system-arm \
  -kernel boot/vmlinuz-4.4.0-38-generic-lpae \
  -initrd boot/initrd.img-4.4.0-38-generic-lpae \
  -append "root=/dev/vda2 rootfstype=ext4" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -monitor telnet:127.0.0.1:9000,server,nowait \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -drive file=ubuntu.img,if=virtio
@sokomo

This comment has been minimized.

sokomo commented Nov 4, 2016

About how to avoiding converting the image, I suggest 2 solutions:

Use raw image

Just create raw image instead of qcow2 image using command:

qemu-img create -f raw ubuntu.img 16G

And then you can continue using without any problem
If you get any warning messages about using raw image:

  • In first qemu-system-arm command, you can replace the -hda ubuntu.img part with -drive format=raw,file=ubuntu,img,index=0.
  • In second qemu-system-arm command, you can replace with format=raw,file=ubuntu.img,if=virtio in -drive option

Note: To avoid using offset when the raw image has multiple partitions, you can reload the loop module:

sudo modprobe -r loop
sudo modprobe loop max_loop=10 max_part=15
sudo losetup -f ubuntu.img
sudo mount /dev/loop0p2 /mnt

After finishing copying the kernel and initrd file, you can un-mount and detach the image file

sudo umount /mnt
sudo losetup -d /dev/loop0

Mount qcow2 image using qemu-nbd

Add nbd module, and then mount using qemu-nbd

sudo modprobe nbd max_part=16
sudo qemu-nbd -c /dev/nbd0 ubuntu.img
sudo partprobe /dev/nbd0
sudo mount /dev/nbd0p2 /mnt

After finishing copying the kernel and initrd file, you can un-mount and detach the image file

sudo umount /mnt/
sudo qemu-nbd -d /dev/nbd0
sudo killall qemu-nbd
@dummys

This comment has been minimized.

dummys commented May 1, 2017

Raw image is not the good idea, because It will take all the space you provide.
If you use LVM in the host, then it's find to use raw image.

@BenGardiner

This comment has been minimized.

BenGardiner commented Jul 12, 2017

re: "TODO: Add working solution that does not require to convert the image file."

Copy the vmlinuz and initrd.img off-of the vm before reboot by using the 'Execute a shell' installer option.

~ # ip route                                                                                                                                                                                                                                                                                                                                                                
default via 10.0.2.2 dev enp0s1                                                                                                                                                                                                                                                                                                                                             
10.0.2.0/24 dev enp0s1  src 10.0.2.15                                                                                                                                                                                                                                                                                                                                       
~ # ping 10.0.2.2                                                                                                                                                                                                                                                                                                                                                           
PING 10.0.2.2 (10.0.2.2): 56 data bytes                                                                                                                                                                                                                                                                                                                                     64 bytes from 10.0.2.2: seq=0 ttl=255 time=8.304 ms                                                                                                                                                                                                                                                                                                                         64 bytes from 10.0.2.2: seq=1 ttl=255 time=9.789 ms                                                                                                                                                                                                                                                                                                                         ^C                                                                                                                                                                                                                                                                                                                                                                          --- 10.0.2.2 ping statistics ---                                                                                                                                                                                                                                                                                                                                            2 packets transmitted, 2 packets received, 0% packet loss                                                                                                                                                                                                                                                                                                                   round-trip min/avg/max = 8.304/9.046/9.789 ms                                                                                                                                                                                                                                                                                                                               
~ # mount                                                                                                                                                                                                                                                                                                                                                                   rootfs on / type rootfs (rw)                                                                                                                                                                                                                                                                                                                                                none on /run type tmpfs (rw,nosuid,relatime,size=101948k,mode=755)                                                                                                                                                                                                                                                                                                          none on /proc type proc (rw,relatime)                                                                                                                                                                                                                                                                                                                                       none on /sys type sysfs (rw,relatime)                                                                                                                                                                                                                                                                                                                                       devtmpfs on /dev type devtmpfs (rw,relatime,size=492152k,nr_inodes=123038,mode=755)                                                                                                                                                                                                                                                                                         devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)                                                                                                                                                                                                                                                                                      /dev/mapper/armhf--xenial--vg-root on /target type ext4 (rw,relatime,errors=remount-ro,data=ordered)                                                                                                                                                                                                                                                                        /dev/vda1 on /target/boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)                                                                                                                                                                                                                                                                                     /dev/mapper/armhf--xenial--vg-root on /dev/.static/dev type ext4 (rw,relatime,errors=remount-ro,data=ordered)                                                                                                                                                                                                                                                               devtmpfs on /target/dev type devtmpfs (rw,relatime,size=492152k,nr_inodes=123038,mode=755)                                                                                                                                                                                                                                                                                  ~ # cd /target/boot/                                                                                                                                                                                                                                                                                                                                                        
/target/boot # nc 10.0.2.2 32008 < vmlinuz                                                                                                                                                                                                                                                                                                                                  
/target/boot # nc 10.0.2.2 32008 < initrd.img                                                                                                                                                                                                                                                                                                                               /target/boot #                                                                                                                                                                                                                                                                                                                                                              

before each nc command above, run a netcat listener on your host; e.g. nc -l 32008 > vmlinuz and nc -l 32008 > initrd.img.

@SeanDHeath

This comment has been minimized.

SeanDHeath commented Feb 8, 2018

You can extract the kernel/initramfs using virt-copy-out from libguestfs. Great tutorial here: https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/

@cpaelzer

This comment has been minimized.

cpaelzer commented Jun 18, 2018

In case you do want to mount partitions instead of offset magic you could use sudo partprobe /dev/loopXX to get proper /dev/loopXXp1, ... entries.

@likan999

This comment has been minimized.

likan999 commented Jul 2, 2018

I followed these steps to install ubuntu18.04 and after installation, vmlinuz appears in /boot but initrd is missing. I retried multiple times and it never worked. Does anyone know why and how to restore the initrd file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment