iscsi / ibft booting test

README.md

iscsi and iBFT test using qemu/kvm and ipxe

The general goal of this doc is to show how to use qemu, ipxe and tgt to test iscsi and iBFT.

There are loads of things that could be done to make this test environment. But for now, it just shows a simple case to show parameters necessary.

Setting up iscsi test environment

In examples here, the iscsi server is available at 192.168.1.10. The qemu host system is available at 192.168.1.11. We assume port 9999 is available to run a python simple web server.

iscsi target setup

If you have an existing iscsi system in place, you can use that. If you need to set one up, this documents how to do so with the tgt package.

• set some variables for later reference Set following variables for later reference and usage. an attempt is provided to get a default.

   export TGT_IMAGES_D=/var/lib/tgt/images
  

• install dependencies

  sudo apt-get update && sudo apt-get install -qy itgt

• add some iscsi target devices

  Ubuntu's /etc/tgt/targets.conf contains include /etc/tgt/conf.d/*.conf, which means you can just drop files into /etc/tgt/conf.d/ and have them read. So to add a target named target-01, just do:

   truncate --size 4G ${TGT_IMAGES_D}/target-01.img
   ./add-tgt-target target-01
   truncate --size 4G ${TGT_IMAGES_D}/target-02.img
   ./add-tgt-target target-02
  

• list targets

  You should now be able to see the targets with both of the following. iscsiadm comes from open-iscsi package.

   sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.10
   sudo tgt-admin --show
  

qemu and ipxe host system setup

• install dependencies

   sudo apt-get update
   sudo apt-get install -qy qemu-system-x86 qemu-utils ipxe
  

• Provide a kernel and initrd

Running Qemu with ipxe

ipxe is a very powerful tool, supporting a lot of functionality. The ipxe package provides an ipxe.lkrn that can be provided to qemu -kernel. The kernel command line then provided is read by ipxe as its config. So, you can do really neat things like:

• Provide 'boot-initrd' and 'boot-kernel'.

  boot-initrd and boot-kernel are a kernel/initrd pair. They can be downloaded from maas.ubuntu.com. If you have open-iscsi installed , likely the files in your /boot will work.

• Run a simple web server to give ipxe the 'config' file.

  This is not strictly necessary, you can put it all inline on the 'append', but this shows a powerful path of receiving configuration from a http endpoint.

   python -m SimpleHTTPServer 9999
  

• Run qemu system

   qemu-system-x86_64 -enable-kvm -m 1024 -curses \
     -device virtio-net-pci,netdev=net00 -netdev type=user,id=net00 \
     -kernel /usr/lib/ipxe/ipxe.lkrn \
     -append "dhcp && chain http://192.168.1.11:9999/config"
  

The system should now boot to initramfs that has a configured iBFT device. See /usr/share/doc/open-iscsi/README.Debian.gz for more information. Once inside, you can look around.

(initramfs) modprobe iscsi_ibft
[   11.444414] iBFT detected.
(initramfs) cat /sys/firmware/ibft/target0/nic-assoc
0
(initramfs) cat /sys/firmware/ibft/target0/target-name
mytest
(initramfs) cat /sys/firmware/ibft/target0/port
3260
(initramfs) cat /sys/firmware/ibft/ethernet0/mac
52:54:00:12:34:56
(initramfs) for x in /sys/class/net/*/address; do echo $x; cat $x; done
/sys/class/net/eth0/address
52:54:00:12:34:56
/sys/class/net/lo/address
00:00:00:00:00:00

$ for f in $(find /sys/firmware/ibft/ -type f); do echo == $f ==; cat $f; done
== /sys/firmware/ibft/target0/lun ==
01000000
== /sys/firmware/ibft/target0/port ==
3260
== /sys/firmware/ibft/target0/target-name ==
inst-000-2
== /sys/firmware/ibft/target0/flags ==
3
== /sys/firmware/ibft/target0/index ==
0
== /sys/firmware/ibft/target0/chap-type ==
0
== /sys/firmware/ibft/target0/nic-assoc ==
0
== /sys/firmware/ibft/target0/ip-addr ==
192.168.1.10
== /sys/firmware/ibft/initiator/flags ==
3
== /sys/firmware/ibft/initiator/index ==
0
== /sys/firmware/ibft/initiator/initiator-name ==
iqn.2010-04.org.ipxe:00000000-0000-0000-0000-000000000000
== /sys/firmware/ibft/ethernet0/mac ==
52:54:00:12:34:56
== /sys/firmware/ibft/ethernet0/vlan ==
0
== /sys/firmware/ibft/ethernet0/flags ==
3
== /sys/firmware/ibft/ethernet0/index ==
0
== /sys/firmware/ibft/ethernet0/primary-dns ==
10.0.2.3
== /sys/firmware/ibft/ethernet0/subnet-mask ==
255.255.255.0
== /sys/firmware/ibft/ethernet0/gateway ==
10.0.2.2
== /sys/firmware/ibft/ethernet0/origin ==
3
== /sys/firmware/ibft/ethernet0/ip-addr ==
10.0.2.15

Limitations

• current ipxe trunk is limited to setting up a single iBFT iscsi target. That limitation is documented on mailing list post. There are for patches available for 'multiple session support' available.

add-tgt-target

#!/bin/sh

Usage() {
   cat <<EOF
Usage: ${0##*/} name [file [read-only]]

   Add a tgt target named 'name' backed by 'file'.
EOF
}

asroot() {
   if [ "$(id -u)" = "0" ]; then
      "$@"
   else
      sudo "$@"
   fi
}

addtarget() {
   local tname="$1" file="$2" ro="$3" fp=""
   [ -z "$2" ] && file="$tname.img"
   if [ ! -f "$file" -a "${file#*/}" != "$file" ] &&
      [ -f "$TGT_IMAGES_D/$file" ]; then
      file="$TGT_IMAGES_D/$file"
   fi
   [ -f "$file" ] || { echo "not a file '$file'"; return 1; }
   asroot sh -c 'tee "${TGT_CONF_D}/${tname}.conf" >/dev/null &&
                 tgt-admin --update ALL' <<EOF
<target ${tname}>
    ${ro:+readonly ${ro}}
    backing-store "$file"
</target>
EOF

TGT_CONF_D=${TGT_CONF_D:-/etc/tgt/conf.d}
TGT_IMAGES_D="${TGT_IMAGES_D:-/var/lib/tgt/images}"
addtarget "$@"

config

#!ipxe
## load this with:
##   -append "dhcp && chain http://192.168.1.131:9999/config"
# shell # uncomment to get an ipxe prompt here
set iscsi-host 192.168.1.10
set base-url http://192.168.1.11:9999/
sanhook --drive 0x80 iscsi:${iscsi-host}::3260:1:target-01
sanhook --drive 0x81 iscsi:${iscsi-host}::3260:1:target-02
kernel ${base-url}/boot-kernel break=top
initrd ${base-url}/boot-initrd
boot

notes.txt

## some setup
pkgs="tgt libvirt-bin qemu-system-x86 qemu-utils ipxe"
sudo apt-get update --quiet
sudo apt-get install --quiet --assume-yes $pkgs </dev/null

##
disk_url="http://cloud-images.ubuntu.com/releases/wily/release/ubuntu-15.10-server-cloudimg-amd64-disk1.img
disk_img="${disk_url##*/}"
disk_raw="${disk_img%.img}.raw"
disk_raw=$(readlink -f "${disk_raw}")

[ -f "$disk_img" ] ||
   { wget "$disk_url" "$disk_img.tmp" && mv "$disk_img.tmp" "$disk_img"; }
[ -f "$disk_raw" ] ||
    { qemu-img convert -O raw "$disk_img" "$disk_raw.tmp" && mv "$disk_raw.tmp" "$disk_raw"; }

## some itgt setup vars
image_dir="$PWD/tgt-images"
mkdir -p "$image_dir"
addtarget() {
   local tname="$1" file="$2" ro="$3"
   [ -z "$2" ] && file="$tname.img"
   [ "${file#/}" = "$file" ] && file="${image_dir}/${file}"
   [ -f "$file" ] || { echo "not a file '$file'"; return 1; }
   sudo tee "/etc/tgt/conf.d/${tname}.conf" <<EOF
<target ${tname}>
    ${ro:+readonly ${ro}}
    backing-store "$file"
</target>
EOF
   sudo tgt-admin --update ALL
}

## set up a tgt target
target_name="mytest"
cp --sparse=always "$disk_raw" "${image_dir}/${target_name}.img"

# patch the cloud image to:
#   a.) resize filesystem so we have some space (bug 1463963)
#   b.) install linux-generic to get iscsi_ibft module. (bug 1511006)
#   c.) install open-iscsi package (bug 1511008)
#        this will include iscsi_ibft if it is available
#   d.) write ISCSI_AUTO=true into /etc/iscsi/iscsi.initramfs 
#       alternatively we could boot with 'iscsi_auto' parameter
#   e.) avoid grub getting updated with root=/dev/nbd0p1
sudo http_proxy=http://nelson:3128 mount-image-callback --system-resolvconf --system-mounts \
  "$image_dir/$target_name.img" -- chroot _MOUNTPOINT_ \
   sh -exc '
      echo workaround bug 1463963
      dev=$(awk "\$2 == \"/\" { d=\$1 }; END { print d }" /proc/mounts)
      resize2fs $dev
      mkdir -p /etc/iscsi && touch /etc/iscsi/iscsi.initramfs
      echo ISCSI_AUTO=true > /etc/iscsi/iscsi.initramfs
      disable_services() {
          local dir="$1"
          cat > "$dir/usr/sbin/policy-rc.d" <<"EOF"
#!/bin/sh
while true; do
    case "$1" in
        -*) shift ;;
        makedev) exit 0 ;;
        x11-common) exit 0 ;;
        *) exit 101 ;;
    esac
done
EOF
          [ $? -eq 0 ] && chmod 755 "$dir/usr/sbin/policy-rc.d" ||
              { error "failed to write policy-rc.d"; return 1; }
      }

      undisable_services() {
          local dir="$1"
          rm -f "$dir/usr/sbin/policy-rc.d"
      }

      disable_services "/"
      trap "undisable_services /" EXIT 
      echo "grub would get the nbd device as root="
      cp /boot/grub/grub.cfg /boot/grub/grub.cfg.save
      apt-get --option=Acquire::Languages=none update &&
      eatmydata apt-get -qy install linux-generic open-iscsi
      cp /boot/grub/grub.cfg.save /boot/grub/grub.cfg
      sync'
          
addtarget "$target_name"
sudo service tgt restart

## now boot it
qemu-system-x86_64 -enable-kvm \
   -device virtio-net-pci,netdev=net00 -netdev type=user,id=net00 \
   -m 1024 -curses -kernel /usr/lib/ipxe/ipxe.lkrn 
   -append 'dhcp && sanhook --drive 0x81 iscsi:192.168.1.131::3260:1:seed && sanboot iscsi:192.168.1.131::3260:1:mytest' 

## this should boot all the way, except for cloud-init looking for a
## datasource which it wont find.
## I did attempt to attach a second drive with cloud-localds on it via
##  'sanhook --drive 0x81 iscsi:192.168.1.131::3260:1:seed'
## but that seemed not to work.  Nothing automatically sets up that target
## during boot.