Skip to content

Instantly share code, notes, and snippets.

@eheydrick
Last active July 8, 2016 21:38
Show Gist options
  • Save eheydrick/ab36a403dd4636aa1ed2ede60647cc60 to your computer and use it in GitHub Desktop.
Save eheydrick/ab36a403dd4636aa1ed2ede60647cc60 to your computer and use it in GitHub Desktop.

Ubuntu 16.04

Having just upgraded from Ubuntu 14.04 Trusty to 16.04 Xenial, I thought I'd share the pain points I encountered.

kswapd uses 100% CPU on smaller instances

There's a bug in newer kernels on EC2 that causes kswapd0 to consume 100% CPU. It seems to happen more often on smaller instances with less RAM (t2.micros for example). There is a workaround changing a udev configuration.

kernel doesn't boot on some larger instances

There was a bug in the 4.4 kernel that prevented booting on larger EC2 instance types such as c3.large and m3.large. This has been fixed in kernel package 4.4.0-24.43.

systemd thread limit

This one bit us hard. In all their wisdom the systemd developers decided to limit the number of threads a process may create to 512. Who would possibly need more than 512 threads?! One of our multithreaded java services quickly hit the limit and stopped working. Those running MySQL hit the problem as well. Thankfully Ubuntu recognized that 512 threads is not enough for many modern applications and removed the limit with systemd package 229-4ubuntu6.

systemd is init

systemd is the init system in 16.04 whereas 14.04 used upstart. Our services were previously started with upstart so we needed to convert from upstart configs to systemd unit files. Thankfully it's not hard to move to systemd. Ubuntu has a nice guide called Systemd for Upstart Users that helps with the transition and I found the systemd man page helpful as well.

apt unattended upgrades enabled by default

There's a feature in Apt to automatically install package updates. This can be useful for automatically installing security updates and keeping your systems up to date. However if you wish to control patching or apply patches some other way unattended-upgrades can get in the way. It was disabled by default in 14.04 but due to a bug in 16.04 it's enabled by default even when you tell it not to. You can disable it by setting APT::Periodic::Unattended-Upgrade "0"; in /etc/apt/apt.conf.d/20auto-upgrades or simply removing that file.

unwanted services on by default

Canonical introduced a number of new features in 16.04 including LXD, their container manager. While some may find the new hotness useful I don't need them in my environment and they enable a bunch of daemons by default. I'm a fan of Just Enough OS images and I like running the minimal set of OS services to support applications. In 16.04 I disable iscsid, open-iscsi, and mdadm services.

LXD mounts interfere with monitoring

LXD adds a bunch of lxcfs mounts of type cgroup. lxcfs is a userspace filesystem for running containers. If you don't care about running containers it's unnecessary. It can also be a problem if you have filesystem monitoring checks that try to check lxcfs mounts and fail because it's not a real filesystem. I had to exclude checking mounts of this type.

pam_mkhomedir not working

I use pam_mkhomedir to create home directories the first time someone logs in. It stopped working in 16.04 for unknown reasons. The fix is to run /bin/sed -i /mkhomedir/d /var/lib/pam/seen followed by /usr/sbin/pam-auth-update --package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment