Skip to content

Instantly share code, notes, and snippets.

Created August 12, 2011 14:52
  • Star 10 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
How to boot up Fedora 15 with systemd inside Libvirt LXC
# Setup a rootfs of Fedora 15 for libvirt lxc
# The rootfs is based on
# See also
# -
if [ $# != 1 ]; then
echo "usage: $0 <hostname>"
exit 1
# Tweak systemd settings to be bootable inside a container
# Set default run level to 3 (multi user)
ln -sf $LIB/ $ETC/
# seems to stick on boot, so disable it. However, we need
# systemd-tmpfiles-setup.service that was started by the dependency of
# to boot up correctly, so start it instead.
cp $LIB/ $ETC/
sed -i 's/' $ETC/
# Stop starting Symlinking one to /dev/null is a standard way
# to disable a target (or a service and others).
ln -s /dev/null $ETC/
# It also a cause of stuck on boot
ln -s /dev/null $ETC/udev-settle.service
# It prevents systemd-tmpfiles-setup.service from starting
ln -s /dev/null $ETC/fedora-readonly.service
# Libvirt lxc provides only tty1
rm -f $ETC/\@tty{2,3,4,5,6}.service
# It launches sulogin on console(tty1) but it interferes getty@tty1
ln -s /dev/null $ETC/console-shell.service
# Workarounds for libvirt 0.9.4. Without this, getty@tty1 doen't launch
# because a trigger event on tty1 doesn't happen.
cp $LIB/getty\@.service $ETC/getty\@.service
sed -i 's/^BindTo/\#&/' $ETC/getty\@.service
ln -sf $ETC/getty\@.service $ETC/\@tty1.service
# Allow a user who logins via ssh to sudo
sed -i 's/^Defaults\ *requiretty/\#&/' /etc/sudoers
# Allow to login at virsh console. doen't work in the absence of auditd
# which cannot run inside a container.
sed -i 's/^.**$/\#&/' /etc/pam.d/login
# Enable eth0 on bootup
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
cat <<EOF > /etc/sysconfig/network
# Tweak sshd configuration
sed -i 's/^UsePAM\ *yes/\#&/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication\ *yes/\#&/' /etc/ssh/sshd_config
# Prevent udevd from creating /dev/ptmx
# This is not needed for libvirt 0.9.5 and above.
cat <<EOF > /etc/udev/rules.d/00-symlink-dev-ptmx.rules
KERNEL=="ptmx", NAME="pts/%k", SYMLINK+="%k"
# Prevent overwriting /dev/pts. Get rid of devpts entry if exists.
sed -i '/devpts/s/^[^#]/\#&/' /etc/fstab
# Allow root to login at virsh console
echo "pts/0" >> /etc/securetty
# Uninstall unnecessary packages of OpenVZ
yum -y remove vzdev vzdummy-init-fc15
# Create a user (if you want)
#adduser -G wheel $USER
#echo "$USER:$PASS" | chpasswd
# Setup host environment for libvirt lxc
# Tested environment
# Host
# - Ubuntu 11.04
# - libvirt 0.9.2 and 0.9.4
# Guest
# - Fedora 15
# Setup a rootfs of Fedora 15
tar zxvf fedora-15-x86_64.tar.gz -C $ROOTFS
# Setup packages. The PPA provides latest libvirt packages.
add-apt-repository ppa:dnjl/build
apt-get update
apt-get install libcgroup1 libvirt0 libvirt-bin
# Allow containers to communicate with outside
echo 1 > /proc/sys/net/ipv4/ip_forward
# Allow to run dhclient of Fedora 15 inside a container
cat - >> /etc/apparmor.d/local/sbin.dhclient <<EOF
/etc/pki/tls/openssl.cnf r,
/var/lib/dhclient/dhclient-*.leases rw,
/var/lib/dhclient/dhclient.leases rw,
apparmor_parser -r /etc/apparmor.d/sbin.dhclient
# Create a container via libvirt
cat - > $XML <<EOF
<domain type='lxc'>
<clock offset='utc'/>
<filesystem type='mount'>
<source dir='$ROOTFS'/>
<target dir='/'/>
<interface type='network'>
<source network='default'/>
<console type='pty' />
virsh -c lxc:/// define $XML
# Setup the rootfs of the container
cp $ROOTFS/tmp
chroot $ROOTFS sh /tmp/ $HOSTNAME
# And have a fun!
#virsh -c lxc:/// start $NAME
#virsh -c lxc:/// console $NAME
# Known issues
# - Networking doesn't work <- FIXED
# - dhclient fails due to AppArmor in the host <- FIXED
# - Many systemd warnings about getty are output periodically <- FIXED
# - Fails to start httpd <- FIXED
# - Cannot login as root at console <- FIXED
# - Cannot login via ssh <- FIXED
# - Loginning via ssh is so slow <- FIXED
# - Many udevd-work errors found in /var/log/message
# - disabling udev.service prevents login console from starting
# - Many /dev/kmsg errors on console <- FIXED
# - /var/lock/subsys/ isn't created correctly <- FIXED
# - It causes that daemons fail to create lock files
# See also
# -
Copy link

peo3 commented Sep 15, 2011

I forgot the template because I've never tried it ;-)

I will give you the template tomorow if you still be interessed
Thanks but I could create a rootfs using the template with a few fixes on Fedora 15 in my container :-)

And I confirmed that a container using the rootfs works well. So the differences between Ubuntu 11.04 host + libvirt and Fedor 15 host + lxc tool cause the different results...

Fedora can use debootstrap and yum... that's why I prefer to work on Fedora... :)
febootstrap should work well like debootstrap :-/

Copy link

peo3 commented Sep 18, 2011

I've also tried Ubuntu 11.04 host + lxc tool with a rootfs which is created by the template and confirmed it doesn't work well; lxc-start doesn't return but processes inside the container seem working. lxc-console doesn't work as well.

Copy link

metal3d commented Sep 23, 2011

This is exactly what I meant. If you have a systemd based "host" that launch a lxc container based on systemd, you will have troubles (very impressives sometimes... xscreensaver is launched, tty are lost, reboot, sometimes X crashes...)

This problem is well known... and I'm seeking a nice way to remove the problem. It seems you have good ideas. So I will continue to try with your script as base, and I'll tell you what's wrong or right :)

Copy link

Hey I have been trying to also launch a f15 container on a f15 host, just applied the configs suggested in the script, X crashed, and container got stuck somewhere, thus unreleasing the cgroup name
peo3 would be of great help if you give me an idea how you did debug those issues

Copy link

metal3d commented Oct 13, 2011

Once again, the problem comes from "systemd"... There is no solution (for now). I'm contacting systemd and lxc administrators to know what should be the solution in the futur :)

Copy link

I understand that the problem comes from systemd, and I would like to know how did you debug it.
systemd has systemd-nspawn which would init a rootfs in a similar way like lxc without most of the features, and i was able to init a f15 rootfs on a f15 host without the crashes. so that tells me something about lxc
Now if we are able to debug and actually find out what goes wrong. so my question is how did you find out about those fixes? how do you debug systemd?

Copy link

metal3d commented Oct 13, 2011

You're right. systemd-nspawn can init a rootfs, and some people say that lxc can be obsolete in the futur. For now, I don't find a way to debug. I only asked some #fedora-fr users (irc on freenode) some questions... It seems to be complicated to debug systemd...

Copy link

peo3 could you hint us on how you debugged systemd?

Copy link

peo3 commented Oct 14, 2011

Hmm. In my case systemd was able to launch without any crashes and it just stuck during service initializations. So debugging is not difficult. I could refer several logs: console log via virsh console, libvirtd error messages from libvirtd log files, syslog, dmesg, and of course systemctl outputs.

Anyway I tried replacing /sbin/init with my fake script /sbin/ which does 'exec /sbin/systemd' at the end of itself. By doing so I could get many information, for example what libvirtd actually deployed, or modify some files, e.g., device files, to see how systemd changes its behavior with the modifications.

I also read the source code of systemd and tried a modified version, although it didn't resolve the problem. (But it's not nothing worth because reading source code corrects my wrong assumption.)

you will have troubles (very impressives sometimes... xscreensaver is launched, tty are lost, reboot, sometimes X crashes...)

I had ever seen the behaviors when I was trying old libvirt. The cause might be that some process was trying to grab /dev/console or /dev/tty*...but I don't remember exactly.

Once again, the problem comes from "systemd"...
Yes. My script may be a workaround and we would need correct fixes in systemd and/or maybe in lxc tool.

BTW, can you try libvirt instead of lxc tool? If using libvirt changes the result, it would give us more hints.

Copy link

xianai commented Mar 10, 2012

Hey, these days I played with lxc and fedora16. I found the same problem of systemd. More specifically, systemd gets a new group /dev/tty_, not those provided by lxc.
LXC starts replacing the /dev/tty1 with a pts (136, 8) while the systemd get the /dev/ of your host machine. So any getty like operation will interfere with your host tty_, which also explains the crash of X. Updated Fedora 16 also has a broken agetty who refuses to cooperate with pts but can be directly replaced by the agetty from debian 6, so that's not a major problem. IMO, it is just insane that systemd pretends to be cgroup aware and can spawn cgroup, but it refuses to reside in a standard spawned cgroup environment, if debian and ubuntu startup daemons also play the trick, then none of them can host each other in containers!

Copy link

InformatiQ commented Mar 10, 2012 via email

Copy link

xianai commented Mar 18, 2012

I simply did this patch, :)

diff -ruNp systemd-37/src/mount-setup.c
--- systemd-37/src/mount-setup.c 2011-08-31 01:21:41.804076227 +0800
+++ 2012-03-11 00:05:55.194903118 +0800
@@ -55,11 +55,6 @@ typedef struct MountPoint {
#define N_EARLY_MOUNT 3

static const MountPoint mount_table[] = {

  •    { "proc",     "/proc",                  "proc",     NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
  •    { "sysfs",    "/sys",                   "sysfs",    NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
  •    { "devtmpfs", "/dev",                   "devtmpfs", "mode=755",          MS_NOSUID,                    true },
  •    { "tmpfs",    "/dev/shm",               "tmpfs",    "mode=1777",         MS_NOSUID|MS_NODEV,           true },
  •    { "devpts",   "/dev/pts",               "devpts",   "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
     { "tmpfs",    "/run",                   "tmpfs",    "mode=755",          MS_NOSUID|MS_NODEV, true },
     { "tmpfs",    "/sys/fs/cgroup",         "tmpfs",    "mode=755",          MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
     { "cgroup",   "/sys/fs/cgroup/systemd", "cgroup",   "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },

after which I found some other very sneaky bugs(does not occur very time, but when it does, it's rather annoying ! ) in login also routed in systemd...
So I just lost my temper with systemd and wrote a very simple init shell script of my own (i. e. simply agetty the lxc tty and start a sshd), so far so good! :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment