Skip to content

Instantly share code, notes, and snippets.

@takeshixx
Last active October 22, 2023 12:14
  • Star 32 You must be signed in to star a gist
  • Fork 14 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save takeshixx/686a4b5e057deff7892913bf69bcb85a to your computer and use it in GitHub Desktop.
Running Ubuntu 16.04.1 armhf on Qemu

Running Ubuntu 16.04.1 armhf on Qemu

This is a writeup about how to install Ubuntu 16.04.1 Xenial Xerus for the 32-bit hard-float ARMv7 (armhf) architecture on a Qemu VM via Ubuntu netboot.

The setup will create a Ubuntu VM with LPAE extensions (generic-lpae) enabled. However, this writeup should also work for non-LPAE (generic) kernels.

The performance of the resulting VM is quite good, and it allows VMs with >1G ram (compared to 256M on versatilepb and 1G on versatile-a9/versatile-a15). It also supports virtio disks whereas versatile-a9/versatile-a15 only support SD cards via the -sd argument.

Get netboot files

The netboot files are available on the official Ubuntu mirror. The following commands will download the kernel (vmlinuz) and initrd (initrd.gz) in a new directory called netboot:

mkdir netboot
cd netboot
wget -r -nH -nd -np -R "index.html*" --quiet http://ports.ubuntu.com/ubuntu-ports/dists/xenial/main/installer-armhf/current/images/generic-lpae/netboot/

Create a image file

The following command creates a image file that will be used as a disk for the Ubuntu system:

qemu-img create -f qcow2 ubuntu.img 16G

Setup networking

Installing Ubuntu via netboot requires an Internet connection. Therefore the Ubuntu VM requires a network intefaces that can use the Internet connection of the host system. The host systems needs to create a tuntap device for the VM and a bridge that will connect the tuntap interface with the Internet-facing interface. The bridge is called br0 and the tuntap device for the Ubuntu VM will be tap0.

sudo ip tuntap add dev tap0 mode tap
sudo ip link set up dev tap0
sudo ip link set tap0 master br0

Creating a bridge and adding the Internet-facing interface (e.g. eth0):

sudo ip link add br0 type bridge
sudo ip link set eth0 master br0

Allow IP packet forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

Start a DHCP server in the bridge so that the Ubuntu VM receives an IP address:

echo "subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.10 192.168.0.100;
  option routers 192.168.0.1;
  option domain-name-servers 208.67.222.222, 208.67.220.220;
}" > qemu-dhcpd.conf
sudo dhcpd -cf qemu-dhcpd.conf br0

Set router IP on bridge interface:

ip addr add 192.168.0.1/24 dev br0
ip link set up dev br0

Start the netboot installation

From the netboot directory (containing the netboot files vmlinuz and initrd.gz), run the following command:

qemu-system-arm \
  -kernel vmlinuz \
  -initrd initrd.gz \
  -append "root=/dev/ram" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -hda ubuntu.img

The kernel should now boot into the Ubuntu installer within the terminal window where the command has been executed.

Extract the new kernel

After the installation process has been finished, extract the kernel and initrd files from the new installed Ubuntu system. Mount the Ubuntu image:

qemu-img convert -f qcow2 -O raw ubuntu.img ubuntu-raw.img
sudo losetup /dev/loop0 ubuntu-raw.img
OFFSET=$(($(sudo fdisk -l /dev/loop0 |grep /dev/loop0p1 |awk '{print $3}')*512))
sudo mount -o loop,offset=$OFFSET /dev/loop0 /mnt

Create a new directory boot for the new kernel and initrd files and copy them into it:

mkdir boot
cp /mnt/initrd.img-4.4.0-38-generic-lpae
cp /mnt/vmlinuz-4.4.0-38-generic-lpae boot

Cleanup:

sudo umount /mnt
sudo losetup -d /dev/loop0
rm ubuntu-raw.img

TODO: Add working solution that does not require to convert the image file.

Start the Ubuntu VM

The Ubuntu VM directory should now have the following structure:

├── boot
│   ├── initrd.img-4.4.0-38-generic-lpae
│   └── vmlinuz-4.4.0-38-generic-lpae
├── netboot
│   ├── initrd.gz
│   └── vmlinuz
└── ubuntu.img

Start Qemu:

qemu-system-arm \
  -kernel boot/vmlinuz-4.4.0-38-generic-lpae \
  -initrd boot/initrd.img-4.4.0-38-generic-lpae \
  -append "root=/dev/vda2 rootfstype=ext4" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -monitor telnet:127.0.0.1:9000,server,nowait \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -drive file=ubuntu.img,if=virtio

The following script can be used for starting the VM:

#!/bin/sh
NET_IF=tap0
BRIDGE=br0
if ! grep --quiet $NET_IF /proc/net/dev;then
    echo "Creating ${NET_IF} device"
    sudo ip tuntap add dev $NET_IF mode tap
    sudo ip l s up dev $NET_IF
fi
if grep --quiet $BRIDGE /proc/net/dev;then
    sudo ip l s $NET_IF master $BRIDGE
else
    echo "${BRIDGE} does not exist!"
    exit
fi
qemu-system-arm \
  -kernel boot/vmlinuz-4.4.0-38-generic-lpae \
  -initrd boot/initrd.img-4.4.0-38-generic-lpae \
  -append "root=/dev/vda2 rootfstype=ext4" \
  -no-reboot \
  -nographic \
  -m 1024 \
  -M virt \
  -serial stdio \
  -monitor telnet:127.0.0.1:9000,server,nowait \
  -net nic \
  -net tap,ifname=tap0,script=no,downscript=no \
  -drive file=ubuntu.img,if=virtio
@sokomo
Copy link

sokomo commented Nov 4, 2016

About how to avoiding converting the image, I suggest 2 solutions:

Use raw image

Just create raw image instead of qcow2 image using command:

qemu-img create -f raw ubuntu.img 16G

And then you can continue using without any problem
If you get any warning messages about using raw image:

  • In first qemu-system-arm command, you can replace the -hda ubuntu.img part with -drive format=raw,file=ubuntu,img,index=0.
  • In second qemu-system-arm command, you can replace with format=raw,file=ubuntu.img,if=virtio in -drive option

Note: To avoid using offset when the raw image has multiple partitions, you can reload the loop module:

sudo modprobe -r loop
sudo modprobe loop max_loop=10 max_part=15
sudo losetup -f ubuntu.img
sudo mount /dev/loop0p2 /mnt

After finishing copying the kernel and initrd file, you can un-mount and detach the image file

sudo umount /mnt
sudo losetup -d /dev/loop0

Mount qcow2 image using qemu-nbd

Add nbd module, and then mount using qemu-nbd

sudo modprobe nbd max_part=16
sudo qemu-nbd -c /dev/nbd0 ubuntu.img
sudo partprobe /dev/nbd0
sudo mount /dev/nbd0p2 /mnt

After finishing copying the kernel and initrd file, you can un-mount and detach the image file

sudo umount /mnt/
sudo qemu-nbd -d /dev/nbd0
sudo killall qemu-nbd

@dummys
Copy link

dummys commented May 1, 2017

Raw image is not the good idea, because It will take all the space you provide.
If you use LVM in the host, then it's find to use raw image.

@BenGardiner
Copy link

re: "TODO: Add working solution that does not require to convert the image file."

Copy the vmlinuz and initrd.img off-of the vm before reboot by using the 'Execute a shell' installer option.

~ # ip route                                                                                                                                                                                                                                                                                                                                                                
default via 10.0.2.2 dev enp0s1                                                                                                                                                                                                                                                                                                                                             
10.0.2.0/24 dev enp0s1  src 10.0.2.15                                                                                                                                                                                                                                                                                                                                       
~ # ping 10.0.2.2                                                                                                                                                                                                                                                                                                                                                           
PING 10.0.2.2 (10.0.2.2): 56 data bytes                                                                                                                                                                                                                                                                                                                                     64 bytes from 10.0.2.2: seq=0 ttl=255 time=8.304 ms                                                                                                                                                                                                                                                                                                                         64 bytes from 10.0.2.2: seq=1 ttl=255 time=9.789 ms                                                                                                                                                                                                                                                                                                                         ^C                                                                                                                                                                                                                                                                                                                                                                          --- 10.0.2.2 ping statistics ---                                                                                                                                                                                                                                                                                                                                            2 packets transmitted, 2 packets received, 0% packet loss                                                                                                                                                                                                                                                                                                                   round-trip min/avg/max = 8.304/9.046/9.789 ms                                                                                                                                                                                                                                                                                                                               
~ # mount                                                                                                                                                                                                                                                                                                                                                                   rootfs on / type rootfs (rw)                                                                                                                                                                                                                                                                                                                                                none on /run type tmpfs (rw,nosuid,relatime,size=101948k,mode=755)                                                                                                                                                                                                                                                                                                          none on /proc type proc (rw,relatime)                                                                                                                                                                                                                                                                                                                                       none on /sys type sysfs (rw,relatime)                                                                                                                                                                                                                                                                                                                                       devtmpfs on /dev type devtmpfs (rw,relatime,size=492152k,nr_inodes=123038,mode=755)                                                                                                                                                                                                                                                                                         devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)                                                                                                                                                                                                                                                                                      /dev/mapper/armhf--xenial--vg-root on /target type ext4 (rw,relatime,errors=remount-ro,data=ordered)                                                                                                                                                                                                                                                                        /dev/vda1 on /target/boot type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)                                                                                                                                                                                                                                                                                     /dev/mapper/armhf--xenial--vg-root on /dev/.static/dev type ext4 (rw,relatime,errors=remount-ro,data=ordered)                                                                                                                                                                                                                                                               devtmpfs on /target/dev type devtmpfs (rw,relatime,size=492152k,nr_inodes=123038,mode=755)                                                                                                                                                                                                                                                                                  ~ # cd /target/boot/                                                                                                                                                                                                                                                                                                                                                        
/target/boot # nc 10.0.2.2 32008 < vmlinuz                                                                                                                                                                                                                                                                                                                                  
/target/boot # nc 10.0.2.2 32008 < initrd.img                                                                                                                                                                                                                                                                                                                               /target/boot #                                                                                                                                                                                                                                                                                                                                                              

before each nc command above, run a netcat listener on your host; e.g. nc -l 32008 > vmlinuz and nc -l 32008 > initrd.img.

@seandheath
Copy link

You can extract the kernel/initramfs using virt-copy-out from libguestfs. Great tutorial here: https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board/

@cpaelzer
Copy link

In case you do want to mount partitions instead of offset magic you could use sudo partprobe /dev/loopXX to get proper /dev/loopXXp1, ... entries.

@likan999
Copy link

likan999 commented Jul 2, 2018

I followed these steps to install ubuntu18.04 and after installation, vmlinuz appears in /boot but initrd is missing. I retried multiple times and it never worked. Does anyone know why and how to restore the initrd file?

@heregoesmarcel
Copy link

I'm experiencing the exact same issue with Ubuntu 18.XX.
Symlinks for initrd are being created but the file itself is missing after installation.

@rfmerrill
Copy link

rfmerrill commented Mar 30, 2019

Re: it not working on 18.04, I've filed this bug against debian-installer: https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1822407

The problem is that the installer thinks you don't need an initrd (probably true for most ARM systems), and virtio_blk is not compiled into the kernel. This seems like an oversight on the part of the installer maintainers.

@rfmerrill
Copy link

rfmerrill commented Mar 31, 2019

Here's how you work around the lack of virtio_blk support, by creating an initrd:

  • Run the installer again by invoking qemu with the same options.
    (Note: on newer versions of qemu you want -serial mon:stdio, as -monitor none -serial stdio will give you an interface where ctrl-c quits the VM immediately, which will be frustrating.)
  • When it gets to the point where it's asking how to partition the disk, select "go back"
  • At the menu it takes you to, select "execute a shell" (or similar, point being you want a shell)
  • Once in the shell:
mkdir /mnt/rootfs
mount /dev/vda2 /mnt/rootfs
mount /dev/vda1 /mnt/rootfs/boot
for dir in /sys /proc /dev; do mount --bind $dir /mnt/rootfs/$dir; done
sudo chroot /mnt/rootfs
bash                                                                 # optional, but ash is a pain to use.
# I needed this because my qemu vm ran very slow, you likely will as well
echo "DefaultTimeoutStartSec=1000s" >> /etc/systemd/system.conf
echo "DefaultTimeoutStopSec=1000s" >> /etc/systemd/system.conf

echo virtio_blk >> /etc/initramfs-tools/modules
sed -i 's/MODULES=most/MODULES=list/' /etc/initramfs-tools/initramfs # MODULES=most will take a LONG time generating a HUGE initrd
ls /boot/vmlinuz-*                                                   # the kernel version string is everything after 'vmlinuz-'
mkinitramfs -v -o /boot/initrd.img-qemu  4.15.0-46-generic-lpae      # insert the correct kernel version here

Exit the chroot and shut the VM down gracefully. The boot partition should now contain the initrd you generated.

I'm running this right now, I'll add a comment to say if it works. Edit: it does!

@rfmerrill
Copy link

Another option might be to cross-compile the same kernel tree that the distro uses with CONFIG_VIRTIO_BLK=y, but I don't think I'm going to trry that.

@Nimamoh
Copy link

Nimamoh commented Apr 22, 2019

I tried everything I could think of but the installer doesn't recognize any network interface. As if the
-net nic \ -net tap,ifname=tap0,script=no,downscript=no \
options were purely ignored.
When going to installer shell, I just have the loop interface when typing ip link
Has anyone had the same issue?

@mohammadhosin
Copy link

to view and extract ubuntu.img
fast way use p7zip app to extract image

@sap-nocops
Copy link

@Nimamoh I have your same issue. Did you find a way to solve it?

@doomedraven
Copy link

doomedraven commented Mar 14, 2020

for networking, this worked for me --netdev type=tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-device,netdev=net0,mac=22:22:22:22:22:22

@tosiara
Copy link

tosiara commented Apr 20, 2020

I had the same problem with missing initrd.img right after installing latest Ubuntu Bionic (both ARM 32 and ARM 64bit)
And the solution from @rfmerrill helped me with a slight change (.conf needed):

sed -i 's/MODULES=most/MODULES=list/' /etc/initramfs-tools/initramfs.conf

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment