Notes about APU2C4 from PC engines
APU2C4 is my main home router. This note is mostly about how to compile and use a custom linux kernel on it, with Alpine Linux.
I like it very much, and (after a year and a half of using it) I still think it was the best choice for a home router/firewall; certainly worth the buck. It's extremely well built (I think), with good choices of components (in both what hardware to put in, and what parts to use). The features that are not that common to come by in other commercial (or open source/hardware for that matter) products which I appreciate a lot are:
- RS232 console port (yay! I don't have to dig up some UART breakout and pinout diagrams to connect to a non-booting Pi or somesuch)
- SATA (so no more USB/hub bottlenecks in hard drive access speed)
- miniPCI express (so my choice of wifi card, and a fast mSATA drive)
- hardware crypto on chip
- x86_64 architecture (easier to deal with than various ARMs)
- ECC RAM!!!
- reasonably good documentation
I use it as a main firewall towards the outside, a wireless access point, and then in containers I have a NAS, logging and processing data from local environmental sensors, a database server, and a http server.
Most of the time the load is close to zero.
Currently I run linux on it (mainly because of the familiarity with the system and the awesome convenience of btrfs, not because I think it's the most secure).
(Don't get me wrong, I like ARM boards a lot, although admittedly more so the microcontroller end of the spectrum; and they give amazing bang for the buck; but... currently there either seems to be good support AND boards not that well suited for a router/firewall purpose (e.g. R-Pi or Beaglebones), OR basically no support (well, one linux distro, i.e. armbian) AND reasonably powerful boards (various Chinese Bananas and FriendlyArms and such). And, the EOL seems to be lightning fast, so maintaining one of these long-term is a pain.)
I run Alpine on my APU2 as the main system (with some other distros set up as LXC containers). The main reason for this is that Alpine can run basically from a ramdisk, the root filesystem is populated on boot with freshly unpacked packages, and then the local modifications are overlayed on this. The local mods are simply saved in a tar.gz file, and versioned. So there is no problem with hard crashes and/or power failures; the setup is easy to refresh and reproduce; and it cleanly boots to the last explicitly saved checkpoint.
I personally have /var on a rw-mounted partition, so some corruption here
is possible.
Of course the stuff that runs in containers (various network services and such) don't enjoy that much protection...
This note is(was?) supposed mainly about custom kernel, so I want to explain how does Alpine Linux deal with kernel in this setup.
Basically, there are three files (which are pretty much independent of the rest
of the system); they reside on the boot partition, usually in the /boot subdir.
They are: vmlinuz-XXX, initramfs-XXX and modloop-XXX. XXX stands for
the kernel flavor, e.g. hardened. Now, the first of these three is the actual
linux kernel, the second one contains the initial ramdisk that the kernel mounts
and runs when booting (which then prepares the main filesystem for Alpine), and
the last file is a squashfs filesystem image with kernel modules and firmware
(it gets mounted to /lib/modules and /lib/firmware).
So if one wants a custom kernel (but keep the rest of the Alpine Linux infrastructure), these three (and only these) need to be taken care of.
Well, the Alpine Linux kernel (as of writing this note) does have one big plus, namely the grsecurity patches. But it also have a couple of minuses as far as the APU2 board is concerned, namely that it doesn't enable ECC, doesn't apply AMD microcode updates, and Alpine doesn't have the correct infrastructure to set the wireless regulatory settings.
And, well, I wanted to be able to tweak the kernel options, and also build all the modules I use into the kernel, so that I can use it to boot into other linux distributions.
The compilation can be run on any linux machine (with the right "development" packages installed). It can be also run on the APU2 itself, and the kernel compilation wasn't as bad as I thought it would be - it took 2.5 hours, with just one core dedicated to the compilation.
I'll use KVER throughout the document to stand for 4.16.5 (which is the
latest kernel version I did this with, and the shared files are for this
version - probably usable with later versions, but not tested and guaranteed).
Get the kernel sources from kernel.org, and unpack them somewhere.
Configure the kernel (make menuconfig or make nconfig). My config (saved
using make savedefconfig) is here (for KVER).
The extra steps to take are:
- to embed the wireless regulatory database, get the
regulatory.dbfile from this official repo and place it where the kernel/compilation_process will find it (with my config it's expected to be in/lib/firmware(yes, absolute path). - the AMD microcode is also expected to be at
/lib/firmware/amd-ucode- I normally get this by installing the distro-appropriate package on the host; usually it's a part oflinux-firmwarepackage. - the above two are configurable with the kernel options
CONFIG_EXTRA_FIRMWAREandCONFIG_EXTRA_FIRMWARE_DIR. - Note: the above wifi stuff applies to
KVER(and presumably above); the earlier kernels had a different mechanism to embed the wireless regulatory file into the kernel (not sure in which kernel version did the change happen). The old notes are in the Rubbish Bin section below. This all is just so that one can avoid using thecrdathing and loading the regulatory stuff on-the-fly, and just be able to useiw reg set GB.
Compile the kernel (make), the result is in arch/x86/boot/bzImage (copy this
to BOOT_PART/DIR/vmlinux-custom or something like that).
I'm not sure this is the "correct" way of using a defconfig file (generated
using make savedefconfig), but here it goes: I copy the defconfig file
to arch/x86/configs/my_defconfig and then run make my_defconfig.
This creates .config which is then used for compilation.
I do use the onboard watchdog, this is CONFIG_SP5100_TCO. However there
apparently are two settings (modules) which prevent initialising the watchdog
when loaded before the sp5100_tco: namely i2c_piix4 and ccp.
The former is disabled in my config, and the latter (a crypto module) is
compiled as a module, just to be able to check if the problem has magically
disappeared (at some point in the future).
The information is from the forums. The recap is that this error:
sp5100_tco: I/O address 0x0cd6 already in use
means that i2c_piix4 needs to be unloaded. The following error:
sp5100_tco: failed to find MMIO address, giving up.
means that ccp needs to be unloaded.
On the other hand, this dmesg output:
sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05
sp5100_tco: PCI Vendor ID: 0x1022, Device ID: 0x780b, Revision ID: 0x42
sp5100_tco: Using 0xfeb00000 for watchdog MMIO address
sp5100_tco: Last reboot was not triggered by watchdog.
sp5100_tco: initialized (0x00000000d5053ddf). heartbeat=60 sec (nowayout=0)
should mean that the hardware watchdog functions properly.
There is CONFIG_LEDS_APU in KVER, but it didn't work for me (I think this bit in dmesg
Unknown PC Engines board: apu2
is from the leds_apu module.)
See below for getting the LEDs working.
kvm-amd now needs to be compiled as a module as well. There is a new
sub-option for kvm-amd which effectively requires the same status as (its
dependency) ccp, even when it's turned off.
The standard command to copy the modules to the right place and depmod them
is make modules_install. However this will copy the modules to
/lib/modules/KERNEL_VERSION, which is not desirable if the goal is to package
them into a modloop file to be used by Alpine.
(Naturally, if you're doing this for use with some other distribution, the
modules should end up in /lib/modules/KERNEL_VERSION on the root filesystem.)
So, to make a modloop file, create a dir modloop alongside linux-VER and
inside linux-VER, run
make INSTALL_MOD_PATH=../modloop modules_install
This will "install" all the in-tree modules (with my config there are just two)
into the correct place inside the modloop directory.
Although the KVER kernel does have an LEDS_APU config parameter, the module
did not work on my machine. There are sources for a kernel module to control
the three LEDs on the APU, originals are here,
linked also from the official support page.
I have a local copy available at local src (see "Binaries" below). Unpack
this alongside the linux-VER directory, cd into it and run make install -
it will compile and install the module into the modloop dir.
There are a couple of options now listed on the official support page. E.g. now there's an "official" userspace library.
However there is also a kernel module which makes the button accessible
as a usual gpio in the sysfs tree, in /sys/class/gpio. I can't find
the original source on the web, I have a local copy at local src, possibly
it's from here. Please note the original
author in the file; I've only updated it so that it compiles with the recent
kernels.
The modloop file is then created by simply making a squashfs filesystem:
mksquashfs modloop/lib modloop-customThe result is called modloop-custom, to be copied alongside vmlinux-custom.
Note that we only create the squashfs image from the subdirs of the lib dir
inside the modloop dir. This is how Alpine expects it to be.
(It does not currently matter what is the name of this file: the Alpine
boot script simply attempts to mount all the files it finds on the boot
partition and checks whether there's modules/$(uname -r) directory in it.)
By the way, if one would want to add some binary firmware blobs to be
available for the kernel, just create modloop/lib/firmware dir and
copy the files in there.
Since pretty much everything (except LEDs support) is compiled in, the initramfs
may not be actually needed. It is needed on Alpine, because it performs
quite a bit more than just loading modules, namely it prepares the root ramdisk.
I currently just use (butchered) Alpine's initramfs, with all the firmware and Alpine kernel's modules removed.
The procedure to do the butchering is:
- create a dir into which it's going to be extracted:
mkdir -p initramfs - copy an original initramfs there
cp /path/to/initramfs-hardened initramfs.orig - change dir:
cd initramfs - extract the existing image (into the current dir):
zcat ../initramfs.orig | cpio -idmv - do the required butchering ... (e.g.
rm -rf lib/{modules,firmware}) - remove the original initramfs:
rm ../initramfs.orig - repack the contents of the current dir as an initramfs:
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs-custom - copy the new initramfs alongside
vmlinux-custom
Add a bootloader entry which loads the new kernel and initramfs. This depends on your bootloader; for syslinux my entry looks like this:
LABEL custom
MENU LABEL Alpine with custom kernel
KERNEL /boot/vmlinux-custom
INITRD /boot/initramfs-custom
APPEND console=ttyS0,115200n8
The three "deploy" files discussed above, compiled by me, can be had at the
deploy files links below. Maybe you find them useful - if your setup is
sufficiently similar to mine, you won't have to compile everything yourself.
Note: I use the uloz.to service to share these (as well as
the local src) files. Below are their sha256 sums, but you should perhaps not
trust any binary files that some random person puts online. The contents of the
local src archive is mostly text files, so you should inspect them.
For the deploy files, you might really want to consider compiling the
things yourself. But of course, I promise that my files have nothing
malicious in them, and they were created exactly as described above.
local src 4.16.5 || deploy files 4.16.5
7e7d9b5d6b105dae40e84d6d5e47679c7891b138ffe4331d8a62cf7a0afa5b5a deploy-4.16.5-custom.tar
8e69185597b863227bc6e78d461e39ef8c7ec366124b27284a70dbc98a48756f src-4.16.5-custom.tar.gz
local src 4.15.1 || deploy files 4.15.1
8ccfd41bf86713798df1f955b4ea439ea2c929a50c0e2f64bc2f2885c6d58a8e deploy-4.15.1-custom.tar
08d1c978d707e6e4ff67ad261cc01813dfc5c45763880bcbc3e6091036626994 src-4.15.1-custom.tar.xz
The old way of embedding wifi-reg-db is not working anymore. The (old) instructions here:
- embedding wireless regdb into the kernel, from: https://github.com/sabotage-linux/sabotage/blob/84a7e90e5851062c6fe4063afd29bba3074c1541/pkg/kernel
- get the text file: https://github.com/sabotage-linux/sabotage/blob/master/KEEP/linux-regdb.txt
- copy it to: net/wireless/db.txt