karlbunch/alpine-zfs-grub-uefi.md

## alpine-zfs-grub-uefi.md

      
    Raw
  

              alpine-zfs-grub-uefi.md
            
          
    Alpine Linux Installation on ZFS Root with grub on UEFI


References [ ref1 | ref2 | ref3 ]


Boot from alpine-extended [ download ]


login as root without any password


setup network interfaces and start networking:
setup-interfaces
/etc/init.d/networking start


add, config, and enable openssh:
apk add openssh
echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
/etc/init.d/sshd start


config root password and show IP so we can ssh in:
echo "root:abc" | chpasswd # set root password to "abc"
ip addr show # show ip address


now we can ssh into the machine


pick a mirror from alpine-mirrors and get the latest version
MIRROR=http://dl-cdn.alpinelinux.org/alpine  # example from mirror above, no trailing slash
ARCH=x86_64 # architecture, 32-bit x86 is "i386" and amd64 is "x86_64"
VERSION=v3.11 # get the latest version from https://alpinelinux.org/ -- do not include last dot


setup apk repos and update:
echo "$MIRROR/$VERSION/main" >> /etc/apk/repositories
echo "$MIRROR/$VERSION/community" >> /etc/apk/repositories
apk update


setup udev to get /dev/disk/by-id
apk add util-linux udev zfs
setup-udev


load zfs module and check version
modprobe zfs      # should return nothing
lsmod | grep zfs  # should list a few modules
zfs --version     # shows version


load sgdisk and dosfstools and list disks:
apk add sgdisk dosfstools
ls -l /dev/disk/by-id  # shows disk by id
ls -l /dev/disk/by-id/ | sed -E 's/^.*[ \t]+([^ \t]+[ \t]+->[ \t]+.*)$/\1/g' # without extraneous info 


lrwxrwxrwx  1 root  root   9 Mar 15 16:44 ata-VBOX_CD-ROM_VB2-01700376 -> ../../sr0
lrwxrwxrwx  1 root  root   9 Mar 15 16:44 ata-VBOX_HARDDISK_VB0cc84e8a-30407bab -> ../../sda
lrwxrwxrwx  1 root  root   9 Mar 15 16:44 ata-VBOX_HARDDISK_VB5ee32bc3-a631fa00 -> ../../sdb


We will use /dev/sda id as $DISK1 and /dev/sdb id as $DISK2 for the following example:
DISK1=/dev/disk/by-id/ata-VBOX_HARDDISK_VB0cc84e8a-30407bab
DISK2=/dev/disk/by-id/ata-VBOX_HARDDISK_VB5ee32bc3-a631fa00


wipe both disks:
sgdisk --zap-all $DISK1
sgdisk --zap-all $DISK2

each command should return:

Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.


create EFI partitions:
sgdisk -n1:1M:+512M -t1:EF00 $DISK1
udevadm settle
mkfs.fat -F 32 -n EFI ${DISK1}-part1
udevadm settle
sgdisk -n1:1M:+512M -t1:EF00 $DISK2
udevadm settle
mkfs.fat -F 32 -n EFI ${DISK2}-part1
udevadm settle


create ZFS partition for boot:
sgdisk -n2:0:+1G -t2:BF00 $DISK1
sgdisk -n2:0:+1G -t2:BF00 $DISK2
udevadm settle


create ZFS partition for root:
sgdisk -n3:0:0 -t3:BF00 $DISK1
sgdisk -n3:0:0 -t3:BF00 $DISK2
udevadm settle


create boot pool (bpool) with features incompatible for grub disabled
BPOOL=bpool
zpool create -o ashift=12 -d \
  -o feature@async_destroy=enabled \
  -o feature@bookmarks=enabled \
  -o feature@embedded_data=enabled \
  -o feature@empty_bpobj=enabled \
  -o feature@enabled_txg=enabled \
  -o feature@extensible_dataset=enabled \
  -o feature@filesystem_limits=enabled \
  -o feature@hole_birth=enabled \
  -o feature@large_blocks=enabled \
  -o feature@lz4_compress=enabled \
  -o feature@spacemap_histogram=enabled \
  -o feature@userobj_accounting=enabled \
  -o feature@zpool_checkpoint=enabled \
  -o feature@spacemap_v2=enabled \
  -o feature@project_quota=enabled \
  -o feature@resilver_defer=enabled \
  -o feature@allocation_classes=enabled \
  -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \
  -O normalization=formD -O relatime=on -O xattr=sa \
  -O mountpoint=none -R /mnt -f $BPOOL mirror ${DISK1}-part2 ${DISK2}-part2


create root pool (use ashift=13 for Samsung SSD):
RPOOL=rpool
zpool create -o ashift=12 \
  -O acltype=posixacl -O canmount=off -O compression=lz4 \
  -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
  -O mountpoint=none -R /mnt \
  -f $RPOOL mirror ${DISK1}-part3 ${DISK2}-part3


create containers for $BPOOL and $RPOOL
zfs create -o mountpoint=none -o canmount=off $BPOOL/BOOT
zfs create -o mountpoint=none -o canmount=off $RPOOL/ROOT


read create your datasets
zfs create -o mountpoint=legacy   -o atime=off $BPOOL/BOOT/default
zfs create -o mountpoint=/        -o atime=off $RPOOL/ROOT/alpine
zfs create -o mountpoint=/home    -o atime=off $RPOOL/HOME
zfs create -o mountpoint=/var/log -o atime=off $RPOOL/LOG


mount boot filesystem
# mount -t zfs $RPOOL/ROOT/alpine  /mnt
mkdir -p /mnt/boot
mount -t zfs $BPOOL/BOOT/default /mnt/boot


Set the bootfs property on the boot filesystem. This command will only be successful when dnodesize property on $BPOOL/BOOT/default is set to legacy:
zfs set dnodesize=legacy $BPOOL/BOOT/default
zpool set bootfs=$BPOOL/BOOT/default $BPOOL


Installing Alpine Linux in a chroot [ ref ]


Find out the latest apk-tools-static version.
echo ${MIRROR}/latest-stable/main/${ARCH}


Go to the above URL and find the file apk-tools-static-?????.apk, As of this writing (2020-03-12) the version is apk-tools-static-2.10.4-r3.apk
Download and unpack the apk-tools-static package to mount directory:
wget ${MIRROR}/latest-stable/main/${ARCH}/apk-tools-static-2.10.4-r3.apk
tar -xzf apk-tools-static-*.apk


Install the alpine base onto chroot
./sbin/apk.static -X ${MIRROR}/latest-stable/main -U --allow-untrusted --root /mnt --initdb add alpine-base


bind mount /dev, /proc, /sys -- see set up the chroot
mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys


copy DNS settings and apk repos to to /mnt
cp /etc/resolv.conf /mnt/etc/resolv.conf
cp /etc/apk/repositories /mnt/etc/apk/repositories


save environment vars for later use:
cat > /mnt/root/vars.sh << EOF
DISK1=$DISK1
DISK2=$DISK2
RPOOL=$RPOOL
BPOOL=$BPOOL
ARCH=$ARCH
MIRROR=$MIRROR
VERSION=$VERSION
EOF


chroot
chroot /mnt /usr/bin/env DISK1=$DISK1 DISK2=$DISK2 RPOOL=$RPOOL BPOOL=$BPOOL /bin/ash -l


mount /dev/cdrom and add necessary stuff
mount /dev/cdrom /media/cdrom
apk add linux-lts udev util-linux zfs-lts openssh nano


perform init process
rc-update add devfs sysinit
rc-update add dmesg sysinit
rc-update add mdev sysinit
rc-update add hwdrivers sysinit

rc-update add hwclock boot
rc-update add modules boot
rc-update add sysctl boot
rc-update add hostname boot
rc-update add bootmisc boot
rc-update add syslog boot

rc-update add mount-ro shutdown
rc-update add killprocs shutdown
rc-update add savecache shutdown


enable zfs services
rc-update add zfs-import sysinit
rc-update add zfs-mount sysinit


Edit the /etc/mkinitfs/mkinitfs.conf file and append nvme zfs module to the features parameter:
# if you don't need nvme: sed -i -E 's/^(features=.*)([^ \t])"$/\1\2 zfs"/g' /etc/mkinitfs/mkinitfs.conf
sed -i -E 's/^(features=.*)([^ \t])"$/\1\2 nvme zfs"/g' /etc/mkinitfs/mkinitfs.conf
cat /etc/mkinitfs/mkinitfs.conf # check the work


rebuild initramfs
mkinitfs $(ls /lib/modules/)


generate fstab for bpool
mv /etc/fstab /etc/fstab.old
cat >> /etc/fstab << EOF
$BPOOL/BOOT/default /boot zfs rw,nodev,noatime,xattr,posixacl 0 0
EOF


generate mount points for EFI partitions:
mkdir -p /efi1
mkdir -p /efi2


add 2 EFI partitions to /etc/fstab
echo UUID=$(blkid -s PARTUUID -o value ${DISK1}-part1 | sed -E 's/^.*[ \t]UUID="([^ \t]+)".*$/\1/g') \
  /efi1 vfat nofail,defaults 0 0 >> /etc/fstab
echo UUID=$(blkid -s PARTUUID -o value ${DISK2}-part1 | sed -E 's/^.*[ \t]UUID="([^ \t]+)".*$/\1/g') \
  /efi2 vfat nofail,defaults 0 0 >> /etc/fstab

echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK1}-part1 | sed -E 's/:.*$//g') \
  /efi1 vfat nofail,defaults 0 0 >> /etc/fstab
echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK2}-part1 | sed -E 's/:.*$//g') \
  /efi2 vfat nofail,defaults 0 0 >> /etc/fstab


check /etc/fstab to be of this format:

bpool/BOOT/default /boot zfs rw,nodev,noatime,xattr,posixacl 0 0
UUID=F857-B91A /boot/efi1 vfat nofail,defaults 0 0
UUID=F889-5FAE /boot/efi2 vfat nofail,defaults 0 0


mount EFI partitions:
mount /efi1
mount /efi2


install grub-efi
apk add efibootmgr grub-efi


grub-probe will report an error. we will fix later


install grub to /efi1
mkdir -p /boot/grub
ZPOOL_VDEV_NAME_PATH=1 grub-probe /boot
# grub-probe should return "zfs"
ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/efi1 --bootloader-id=GRUB
#ZPOOL_VDEV_NAME_PATH=1 grub-install --target=x86_64-efi --efi-directory=/boot/efi2 --bootloader-id=GRUB2


modify /etc/default/grub - add root=ZFS=$RPOOL/ROOT/alpine to GRUB_CMDLINE_LINUX
[[ -f /etc/default/grub.original ]] && cp /etc/default/grub.original /etc/default/grub
cp /etc/default/grub /etc/default/grub.original
# add GRUB_CMDLINE_LINUX="root=$RPOOL/ROOT/alpine"
echo "GRUB_CMDLINE_LINUX=\"root=""$RPOOL""/ROOT/alpine rootfstype=zfs\"" >> /etc/default/grub
# add stuff to to GRUB_CMDLINE_LINUX_DEFAULT
echo "GRUB_CMDLINE_LINUX_DEFAULT=\"modules=sd-mod,usb-storage,ext4 nomoodeset\"" >> /etc/default/grub
# remove crap from GRUB_CMDLINE_LINUX_DEFAULT
# sed -i -E 's/^(GRUB_CMDLINE_LINUX_DEFAULT=).*$/\1=""/g' /etc/default/grub


make new /boot/grub/grub.cfg
ZPOOL_VDEV_NAME_PATH=1 grub-mkconfig -o /boot/grub/grub.cfg


it will report an error. BUMMER 2 choices below


run script to fix what's manually done in 2.

cp /boot/grub/grub.cfg /boot/grub/grub.cfg.original
sed -i -E 's;^([ \t]*linux[ \t]+/BOOT/default@/vmlinuz[^ \t]*[ \t]+root=).*$;\1'"$RPOOL"'/ROOT/alpine rootfstype=zfs modules=sd-mod,usb-storage,ext4 nomodeset;g' /boot/grub/grub.cfg
sed -i -E '/^\/dev\/sd[a-z]3[ \t]+ro[ \t]+root='"$RPOOL"'\/ROOT\/alpine.*$/d' /boot/grub/grub.cfg


Manually fix it. edit near line "echo 'Loading Linux lts ...' followed by linux /BOOT/default@/vmlinuz... so it reads:

echo 'Loading Linux lts ...'
linux /BOOT/default@/vmlinuz-lts rootfstype=zfs root=$RPOOL/ROOT/alpine ro modules=sd-mod,usb-storage,ext4 nomodeset 
echo 'Loading initial ramdisk ...'


rsync both partitions to make both bootable
apk add rsync
rsync -Rai --stats --human-readable --delete --verbose --progress /efi1/./ /efi2


add EFI boot entry to machine
efibootmgr -c -g -d $DISK2 -p 1 -L "GRUB-2" -l '\EFI\GRUB\grubx64.efi'


(optional) disable media use for apk installation:
sed -i -E 's/^(\/media.*)$/#\1/g' /etc/apk/repositories


(optional) permit root login on ssh:
sed -i -E 's/^(#PermitRootLogin.*)$/PermitRootLogin yes\n\1/g' /etc/ssh/sshd_config


(optional-required for ssh login) set root password:
echo "root:abc" | chpasswd


set ZFS pool cachefile:
zpool set cachefile=/etc/zfs/zpool.cache $RPOOL


exit chroot
exit


Done with chroot, prepare to reboot


unmount all
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}


unmount zfs and export pools
umount /mnt/boot
zfs unmount $RPOOL/LOG
zfs unmount $RPOOL/HOME
zfs unmount -a


remove live USB/CD and reboot
# remove live media
reboot


Post Installation Setup


login as root with or without password (depending on whether you set root password)
setup networking:
setup-interfaces
/etc/init.d/networking restart


Hopefully ethernet access will be on: try ping -c 3 google.com
If you haven't enabled sshd or set root password, go back and do it:
ssh login to machine with root and password
can mount live media for faster apk updates:
mount /dev/cdrom /media/cdrom


fix grub-efi install error from earlier:
cp /boot/grub/grub.cfg /boot/grub/grub.cfg.working
apk fix # this will fix grub-efi error
cp /boot/grub/grub.cfg.working /boot/grub/grub.cfg


add some useful software:
apk add which htop man man-pages


setup the rest of alpine
# cp /etc/apk/repositories /etc/apk/repositories.save
setup-udev
setup-keymap
setup-hostname
setup-timezone
setup-proxy
setup-ntp
# setup-apkrepos
# setup-sshd
setup-ntp # use chrony
# setup-alpine # when asked "which disk to use?" answer "none"
# cp /etc/apk/repositories.save /etc/apk/repositories # don't wanna mess with repo file


change root to run bash instead of ash:
apk add bash bash-completion shadow sudo
usermod -s /bin/bash root


logout and login again to take effect


add another user
useradd -s /bin/bash -m -U username
echo "username:1234" | chpasswd


add that user to wheel group so user can sudo
EDITOR=nano visudo


uncomment either  a) # %wheel ALL=(ALL) ALL or b) # %wheel ALL=(ALL) NOPASSWD: ALL
if you uncomment a), password is needed for sudo, if you uncomment b), password is not needed for sudo
add a user to wheel group so that user can sudo
usermod -aG wheel username


login as a non-root user and use it to disable direct root login:
sudo passwd -l root

(sudo passwd -u root will unlock root access)
reboot and login as root