Skip to content

Instantly share code, notes, and snippets.

Created August 12, 2011 14:52
  • Star 10 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save peo3/1142202 to your computer and use it in GitHub Desktop.
How to boot up Fedora 15 with systemd inside Libvirt LXC
# Setup a rootfs of Fedora 15 for libvirt lxc
# The rootfs is based on
# See also
# -
if [ $# != 1 ]; then
echo "usage: $0 <hostname>"
exit 1
# Tweak systemd settings to be bootable inside a container
# Set default run level to 3 (multi user)
ln -sf $LIB/ $ETC/
# seems to stick on boot, so disable it. However, we need
# systemd-tmpfiles-setup.service that was started by the dependency of
# to boot up correctly, so start it instead.
cp $LIB/ $ETC/
sed -i 's/' $ETC/
# Stop starting Symlinking one to /dev/null is a standard way
# to disable a target (or a service and others).
ln -s /dev/null $ETC/
# It also a cause of stuck on boot
ln -s /dev/null $ETC/udev-settle.service
# It prevents systemd-tmpfiles-setup.service from starting
ln -s /dev/null $ETC/fedora-readonly.service
# Libvirt lxc provides only tty1
rm -f $ETC/\@tty{2,3,4,5,6}.service
# It launches sulogin on console(tty1) but it interferes getty@tty1
ln -s /dev/null $ETC/console-shell.service
# Workarounds for libvirt 0.9.4. Without this, getty@tty1 doen't launch
# because a trigger event on tty1 doesn't happen.
cp $LIB/getty\@.service $ETC/getty\@.service
sed -i 's/^BindTo/\#&/' $ETC/getty\@.service
ln -sf $ETC/getty\@.service $ETC/\@tty1.service
# Allow a user who logins via ssh to sudo
sed -i 's/^Defaults\ *requiretty/\#&/' /etc/sudoers
# Allow to login at virsh console. doen't work in the absence of auditd
# which cannot run inside a container.
sed -i 's/^.**$/\#&/' /etc/pam.d/login
# Enable eth0 on bootup
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
cat <<EOF > /etc/sysconfig/network
# Tweak sshd configuration
sed -i 's/^UsePAM\ *yes/\#&/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication\ *yes/\#&/' /etc/ssh/sshd_config
# Prevent udevd from creating /dev/ptmx
# This is not needed for libvirt 0.9.5 and above.
cat <<EOF > /etc/udev/rules.d/00-symlink-dev-ptmx.rules
KERNEL=="ptmx", NAME="pts/%k", SYMLINK+="%k"
# Prevent overwriting /dev/pts. Get rid of devpts entry if exists.
sed -i '/devpts/s/^[^#]/\#&/' /etc/fstab
# Allow root to login at virsh console
echo "pts/0" >> /etc/securetty
# Uninstall unnecessary packages of OpenVZ
yum -y remove vzdev vzdummy-init-fc15
# Create a user (if you want)
#adduser -G wheel $USER
#echo "$USER:$PASS" | chpasswd
# Setup host environment for libvirt lxc
# Tested environment
# Host
# - Ubuntu 11.04
# - libvirt 0.9.2 and 0.9.4
# Guest
# - Fedora 15
# Setup a rootfs of Fedora 15
tar zxvf fedora-15-x86_64.tar.gz -C $ROOTFS
# Setup packages. The PPA provides latest libvirt packages.
add-apt-repository ppa:dnjl/build
apt-get update
apt-get install libcgroup1 libvirt0 libvirt-bin
# Allow containers to communicate with outside
echo 1 > /proc/sys/net/ipv4/ip_forward
# Allow to run dhclient of Fedora 15 inside a container
cat - >> /etc/apparmor.d/local/sbin.dhclient <<EOF
/etc/pki/tls/openssl.cnf r,
/var/lib/dhclient/dhclient-*.leases rw,
/var/lib/dhclient/dhclient.leases rw,
apparmor_parser -r /etc/apparmor.d/sbin.dhclient
# Create a container via libvirt
cat - > $XML <<EOF
<domain type='lxc'>
<clock offset='utc'/>
<filesystem type='mount'>
<source dir='$ROOTFS'/>
<target dir='/'/>
<interface type='network'>
<source network='default'/>
<console type='pty' />
virsh -c lxc:/// define $XML
# Setup the rootfs of the container
cp $ROOTFS/tmp
chroot $ROOTFS sh /tmp/ $HOSTNAME
# And have a fun!
#virsh -c lxc:/// start $NAME
#virsh -c lxc:/// console $NAME
# Known issues
# - Networking doesn't work <- FIXED
# - dhclient fails due to AppArmor in the host <- FIXED
# - Many systemd warnings about getty are output periodically <- FIXED
# - Fails to start httpd <- FIXED
# - Cannot login as root at console <- FIXED
# - Cannot login via ssh <- FIXED
# - Loginning via ssh is so slow <- FIXED
# - Many udevd-work errors found in /var/log/message
# - disabling udev.service prevents login console from starting
# - Many /dev/kmsg errors on console <- FIXED
# - /var/lock/subsys/ isn't created correctly <- FIXED
# - It causes that daemons fail to create lock files
# See also
# -
Copy link

peo3 commented Oct 14, 2011

Hmm. In my case systemd was able to launch without any crashes and it just stuck during service initializations. So debugging is not difficult. I could refer several logs: console log via virsh console, libvirtd error messages from libvirtd log files, syslog, dmesg, and of course systemctl outputs.

Anyway I tried replacing /sbin/init with my fake script /sbin/ which does 'exec /sbin/systemd' at the end of itself. By doing so I could get many information, for example what libvirtd actually deployed, or modify some files, e.g., device files, to see how systemd changes its behavior with the modifications.

I also read the source code of systemd and tried a modified version, although it didn't resolve the problem. (But it's not nothing worth because reading source code corrects my wrong assumption.)

you will have troubles (very impressives sometimes... xscreensaver is launched, tty are lost, reboot, sometimes X crashes...)

I had ever seen the behaviors when I was trying old libvirt. The cause might be that some process was trying to grab /dev/console or /dev/tty*...but I don't remember exactly.

Once again, the problem comes from "systemd"...
Yes. My script may be a workaround and we would need correct fixes in systemd and/or maybe in lxc tool.

BTW, can you try libvirt instead of lxc tool? If using libvirt changes the result, it would give us more hints.

Copy link

xianai commented Mar 10, 2012

Hey, these days I played with lxc and fedora16. I found the same problem of systemd. More specifically, systemd gets a new group /dev/tty_, not those provided by lxc.
LXC starts replacing the /dev/tty1 with a pts (136, 8) while the systemd get the /dev/ of your host machine. So any getty like operation will interfere with your host tty_, which also explains the crash of X. Updated Fedora 16 also has a broken agetty who refuses to cooperate with pts but can be directly replaced by the agetty from debian 6, so that's not a major problem. IMO, it is just insane that systemd pretends to be cgroup aware and can spawn cgroup, but it refuses to reside in a standard spawned cgroup environment, if debian and ubuntu startup daemons also play the trick, then none of them can host each other in containers!

Copy link

InformatiQ commented Mar 10, 2012 via email

Copy link

xianai commented Mar 18, 2012

I simply did this patch, :)

diff -ruNp systemd-37/src/mount-setup.c
--- systemd-37/src/mount-setup.c 2011-08-31 01:21:41.804076227 +0800
+++ 2012-03-11 00:05:55.194903118 +0800
@@ -55,11 +55,6 @@ typedef struct MountPoint {
#define N_EARLY_MOUNT 3

static const MountPoint mount_table[] = {

  •    { "proc",     "/proc",                  "proc",     NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
  •    { "sysfs",    "/sys",                   "sysfs",    NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
  •    { "devtmpfs", "/dev",                   "devtmpfs", "mode=755",          MS_NOSUID,                    true },
  •    { "tmpfs",    "/dev/shm",               "tmpfs",    "mode=1777",         MS_NOSUID|MS_NODEV,           true },
  •    { "devpts",   "/dev/pts",               "devpts",   "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
     { "tmpfs",    "/run",                   "tmpfs",    "mode=755",          MS_NOSUID|MS_NODEV, true },
     { "tmpfs",    "/sys/fs/cgroup",         "tmpfs",    "mode=755",          MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
     { "cgroup",   "/sys/fs/cgroup/systemd", "cgroup",   "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },

after which I found some other very sneaky bugs(does not occur very time, but when it does, it's rather annoying ! ) in login also routed in systemd...
So I just lost my temper with systemd and wrote a very simple init shell script of my own (i. e. simply agetty the lxc tty and start a sshd), so far so good! :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment