Skip to content

Instantly share code, notes, and snippets.

@peo3

peo3/setup_lxc_rootfs_fedora15.sh

Created August 12, 2011 14:52
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save peo3/1142202 to your computer and use it in GitHub Desktop.
Save peo3/1142202 to your computer and use it in GitHub Desktop.
Download ZIP
How to boot up Fedora 15 with systemd inside Libvirt LXC
Raw
setup_lxc_rootfs_fedora15.sh
#!/bin/sh
# Setup a rootfs of Fedora 15 for libvirt lxc
#
# The rootfs is based on http://download.openvz.org/template/precreated/fedora-15-x86_64.tar.gz
#
# See also
# - http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01707/lxc-fedora.in
if [ $# != 1 ]; then
echo "usage: $0 <hostname>"
exit 1
fi
HOSTNAME=$1
#
# Tweak systemd settings to be bootable inside a container
#
ETC=/etc/systemd/system
LIB=/lib/systemd/system
# Set default run level to 3 (multi user)
ln -sf $LIB/multi-user.target $ETC/default.target
# sysinit.target seems to stick on boot, so disable it. However, we need
# systemd-tmpfiles-setup.service that was started by the dependency of
# sysinit.target to boot up correctly, so start it instead.
cp $LIB/basic.target $ETC/basic.target
sed -i 's/sysinit.target/systemd-tmpfiles-setup.service/' $ETC/basic.target
# Stop starting sysinit.target. Symlinking one to /dev/null is a standard way
# to disable a target (or a service and others).
ln -s /dev/null $ETC/sysinit.target
# It also a cause of stuck on boot
ln -s /dev/null $ETC/udev-settle.service
# It prevents systemd-tmpfiles-setup.service from starting
ln -s /dev/null $ETC/fedora-readonly.service
# Libvirt lxc provides only tty1
rm -f $ETC/getty.target.wants/getty\@tty{2,3,4,5,6}.service
# It launches sulogin on console(tty1) but it interferes getty@tty1
ln -s /dev/null $ETC/console-shell.service
#
# Workarounds for libvirt 0.9.4. Without this, getty@tty1 doen't launch
# because a trigger event on tty1 doesn't happen.
#
cp $LIB/getty\@.service $ETC/getty\@.service
sed -i 's/^BindTo/\#&/' $ETC/getty\@.service
ln -sf $ETC/getty\@.service $ETC/getty.target.wants/getty\@tty1.service
# Allow a user who logins via ssh to sudo
sed -i 's/^Defaults\ *requiretty/\#&/' /etc/sudoers
#
# Allow to login at virsh console. loginuid.so doen't work in the absence of auditd
# which cannot run inside a container.
#
sed -i 's/^.*loginuid.so.*$/\#&/' /etc/pam.d/login
#
# Enable eth0 on bootup
#
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
EOF
cat <<EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=$HOSTNAME
EOF
#
# Tweak sshd configuration
#
sed -i 's/^UsePAM\ *yes/\#&/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication\ *yes/\#&/' /etc/ssh/sshd_config
#
# Prevent udevd from creating /dev/ptmx
# This is not needed for libvirt 0.9.5 and above.
#
cat <<EOF > /etc/udev/rules.d/00-symlink-dev-ptmx.rules
KERNEL=="ptmx", NAME="pts/%k", SYMLINK+="%k"
EOF
#
# Prevent overwriting /dev/pts. Get rid of devpts entry if exists.
#
sed -i '/devpts/s/^[^#]/\#&/' /etc/fstab
#
# Allow root to login at virsh console
#
echo "pts/0" >> /etc/securetty
#
# Uninstall unnecessary packages of OpenVZ
#
yum -y remove vzdev vzdummy-init-fc15
#
# Create a user (if you want)
#
#USER=peo3
#PASS=peo3pass
#adduser -G wheel $USER
#echo "$USER:$PASS" | chpasswd
Raw
setup_ubuntu1104_for_libvirt_lxc.sh
#!/bin/sh
# Setup host environment for libvirt lxc
#
# Tested environment
#
# Host
# - Ubuntu 11.04
# - libvirt 0.9.2 and 0.9.4
# Guest
# - Fedora 15 http://download.openvz.org/template/precreated/fedora-15-x86_64.tar.gz
ROOTFS=/opt/rootfs/fedora15
NAME=fedora15
HOSTNAME=$NAME
#
# Setup a rootfs of Fedora 15
#
wget http://download.openvz.org/template/precreated/fedora-15-x86_64.tar.gz
tar zxvf fedora-15-x86_64.tar.gz -C $ROOTFS
#
# Setup packages. The PPA provides latest libvirt packages.
#
add-apt-repository ppa:dnjl/build
apt-get update
apt-get install libcgroup1 libvirt0 libvirt-bin
#
# Allow containers to communicate with outside
#
echo 1 > /proc/sys/net/ipv4/ip_forward
#
# Allow to run dhclient of Fedora 15 inside a container
#
cat - >> /etc/apparmor.d/local/sbin.dhclient <<EOF
/etc/pki/tls/openssl.cnf r,
/var/lib/dhclient/dhclient-*.leases rw,
/var/lib/dhclient/dhclient.leases rw,
EOF
apparmor_parser -r /etc/apparmor.d/sbin.dhclient
#
# Create a container via libvirt
#
XML=/tmp/fedora15.xml
cat - > $XML <<EOF
<domain type='lxc'>
<name>$NAME</name>
<memory>128000</memory>
<os>
<type>exe</type>
<init>/sbin/init</init>
</os>
<vcpu>1</vcpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
<filesystem type='mount'>
<source dir='$ROOTFS'/>
<target dir='/'/>
</filesystem>
<interface type='network'>
<source network='default'/>
</interface>
<console type='pty' />
</devices>
</domain>
EOF
virsh -c lxc:/// define $XML
#
# Setup the rootfs of the container
#
cp setup_lxc_rootfs_fedora15.sh $ROOTFS/tmp
chroot $ROOTFS sh /tmp/setup_lxc_rootfs_fedora15.sh $HOSTNAME
# And have a fun!
#virsh -c lxc:/// start $NAME
#virsh -c lxc:/// console $NAME
# Known issues
# - Networking doesn't work <- FIXED
# - dhclient fails due to AppArmor in the host <- FIXED
# - Many systemd warnings about getty are output periodically <- FIXED
# - Fails to start httpd <- FIXED
# - Cannot login as root at console <- FIXED
# - Cannot login via ssh <- FIXED
# - Loginning via ssh is so slow <- FIXED
# - Many udevd-work errors found in /var/log/message
# - disabling udev.service prevents login console from starting
# - Many /dev/kmsg errors on console <- FIXED
# - /var/lock/subsys/ isn't created correctly <- FIXED
# - It causes that daemons fail to create lock files
#
# See also
# - http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01707/lxc-fedora.in
@xianai
Copy link

xianai commented Mar 10, 2012

Hey, these days I played with lxc and fedora16. I found the same problem of systemd. More specifically, systemd gets a new group /dev/tty_, not those provided by lxc.
LXC starts replacing the /dev/tty1 with a pts (136, 8) while the systemd get the /dev/ of your host machine. So any getty like operation will interfere with your host tty_, which also explains the crash of X. Updated Fedora 16 also has a broken agetty who refuses to cooperate with pts but can be directly replaced by the agetty from debian 6, so that's not a major problem. IMO, it is just insane that systemd pretends to be cgroup aware and can spawn cgroup, but it refuses to reside in a standard spawned cgroup environment, if debian and ubuntu startup daemons also play the trick, then none of them can host each other in containers!

@InformatiQ
Copy link

InformatiQ commented Mar 10, 2012 via email

@xianai
Copy link

xianai commented Mar 18, 2012

I simply did this patch, :)

diff -ruNp systemd-37/src/mount-setup.c systemd-37.new/src/mount-setup.c
--- systemd-37/src/mount-setup.c 2011-08-31 01:21:41.804076227 +0800
+++ systemd-37.new/src/mount-setup.c 2012-03-11 00:05:55.194903118 +0800
@@ -55,11 +55,6 @@ typedef struct MountPoint {
#define N_EARLY_MOUNT 3

static const MountPoint mount_table[] = {

  •    { "proc",     "/proc",                  "proc",     NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },

  •    { "sysfs",    "/sys",                   "sysfs",    NULL,                MS_NOSUID|MS_NOEXEC|MS_NODEV, true },

  •    { "devtmpfs", "/dev",                   "devtmpfs", "mode=755",          MS_NOSUID,                    true },

  •    { "tmpfs",    "/dev/shm",               "tmpfs",    "mode=1777",         MS_NOSUID|MS_NODEV,           true },

  •    { "devpts",   "/dev/pts",               "devpts",   "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
 { "tmpfs",    "/run",                   "tmpfs",    "mode=755",          MS_NOSUID|MS_NODEV, true },
 { "tmpfs",    "/sys/fs/cgroup",         "tmpfs",    "mode=755",          MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
 { "cgroup",   "/sys/fs/cgroup/systemd", "cgroup",   "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },

after which I found some other very sneaky bugs(does not occur very time, but when it does, it's rather annoying ! ) in login also routed in systemd...
So I just lost my temper with systemd and wrote a very simple init shell script of my own (i. e. simply agetty the lxc tty and start a sshd), so far so good! :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment