Systemd template unit for controlling Cloud-hypervisor guests

ch@.service

[Unit]
Description=Cloud-Hypervisor for %i
After=network.target
After=local-fs.target
After=remote-fs.target
After=systemd-logind.service
After=systemd-machined.service
Wants=network.target

[Service]
SyslogLevel=debug
Type=simple
LogsDirectory=ch
StateDirectory=ch
StandardOutput=append:/var/log/ch/%i.stdout
WorkingDirectory=/var/lib/ch
RuntimeDirectory=ch
RuntimeDirectoryMode=0775
RuntimeDirectoryPreserve=yes
EnvironmentFile=/var/lib/ch/%i.env
ExecStartPre=/bin/bash -c 'for IF in ${CH_BRS};do ip l sh ${IF%:*} >/dev/null 2>&1 || (ip tuntap add ${IF%:*} mode tap && ip l s ${IF%:*} master ${IF#*:});done'
ExecStart=/bin/bash -c 'cloud-hypervisor --api-socket $RUNTIME_DIRECTORY/%i-sock ${CH_CONFIG}'
ExecStop=/bin/bash -c 'ch-remote --api-socket $RUNTIME_DIRECTORY/%i-sock shutdown-vmm'
ExecStop=/bin/bash -c 'for IF in ${CH_BRS};do ip l sh ${IF%:*} >/dev/null 2>&1 && ip l del ${IF%:*};done'
#ExecStop=/bin/bash -c 'rm -fv $RUNTIME_DIRECTORY/%i-sock'
ExecReload=/bin/bash -c 'ch-remote --api-socket $RUNTIME_DIRECTORY/%i-sock reboot'

[Install]
WantedBy=multi-user.target

create-cloud-init.sh

              
#!/bin/bash                                                    
set -e                                                         
#set -x                     
#[[ "$EUID" -ne 0 ]] && echo "Please run as root" && exit    
# ./create-cloud-init.sh trax 44:c1 10.0.1.41/26                               
usage(){                                                       
echo "Usage: $0 [name] [mac] [ip] [gw] [nssearch] [nsaddr] " 
}                                      
[[ $# -lt 3 ]] && (usage && exit)                                              
[[ $1 == *.* ]] && (echo "No dots in arg1" && usage && exit)                   
                                                               
_dir="/var/lib/ch"
vm="$1"        
c_init="${_dir}/${vm}-init.img"                                                
#set -x                                                                                                                       
bridge="brvirt"                                                
                                                               
[[ $2 == *:* ]] && mac="22:22:22:22:$2" || mac="22:22:22:22:aa:a1"                                                                                            
[[ $3 == *.*.*.* ]] && ip="$3" || ip="10.0.1.60/26"    
[[ $4 == *.*.*.* ]] && gw="$4" || gw="10.0.1.1"                                
[[ $5 == *.* ]] && nssearch="$5" || nssearch="h3m,h3m.li"                      
[[ $6 == *.*.*.* ]] && nsaddr="$6" || nsaddr="10.0.1.5"                        
                               
gen_nw_cfg(){
nw_cfg="version: 2
ethernets:            
  ens2:              
    match:      
      macaddress: "$1"
    addresses: ["$2"]  
    nameservers:
      search: ["$4"]
      addresses: ["$5"]
    routes:
      - to: default                                            
        via: "$3"
"                                                              
printf "${nw_cfg}" > network-config                                            
}                                                              
gen_nw_cfg $mac $ip $gw $nssearch $nsaddr                                      
                               
printf "CH_CONFIG=--kernel ./hypervisor-fw \   
--cpus boot=2 \             
--memory size=2G,shared=on \
--disk path=$vm.raw --disk path=${c_init##*/} \
--net tap=${vm},mac=${mac} \
--serial tty --console off \
--log-file /var/log/ch/${vm}.log -v                                            
CH_BRS="${vm}:${bridge}"                                       
" > "${_dir}/${vm}.env"                
                                                                                                                              
printf "instance-id: $vm \nlocal-hostname: $vm\n" > meta-data

## cloud-localds -v -H $vm -N network-config.${vm} $c_init user-data.${vm}                                                                                    
[[ -f "${c_init}" ]] && sudo rm -vf "${c_init}"                                
#mkdosfs -n CIDATA -C "${c_init}" 4096 > /dev/null                             
mkdosfs -n CIDATA -C "${c_init}" 64 > /dev/null                                
mcopy -oi "${c_init}" -s user-data ::                                          
mcopy -oi "${c_init}" -s meta-data ::                                          
mcopy -oi "${c_init}" -s network-config ::                                     

cat meta-data network-config           
rm -f meta-data network-config         

example.env

CH_CONFIG=--kernel ./hypervisor-fw --cpus boot=2 --memory size=8G,shared=on --disk path=px7.raw --net tap=px7,mac=22:22:22:14:fa:a0 --serial tty --console off --log-file /var/log/ch/px7.log -v
CH_BRS=px7:brvirt

@mortn love it! I have a few things here; https://github.com/artificialwisdomai/origin/tree/main/platform/node

I also will paste a few of my (definately not done definately work in progress definately doesn't work scripts).

The unit file:

sdake@beast-06:/usr/lib/systemd/system$ cat sdacctl@.service
[Unit]
Description=Artificial Wisdom™ SDAC™ core %I
ConditionPathExists=/etc/sdacctl.d/%I

[Service]
StateDirectory=/var/lib/artificialwisdom/%I
RuntimeDirectory=/run/artificialwisdom/%I
ExecStart=/usr/local/bin/cloud-hypervisor --kernel $STATE_DIRECTORY/hypervisor-fw --disk path=$STATE_DIRECTORY/baseline.img,direct=on --disk path=$STATE_DIRECTORY/cloudinit_conifg.img,readonly=on,direct=on --api-socket $RUNTIME_DIRECTORY/api.sock --vsock cid=3,socket=$RUNTIME_DIRECTORY/vsock --serial tty --console pty --cpus boot=32 --net fd=3,mac=${mac} -v --cmdline rd.module_blacklist=nouveau,nvidiafb console=tty0 root=/dev/vda1 rw --memory size=128G,hugepages=on,hugepage_size=2M --device path=/sys/bus/pci/devices/0000:%I:00.0 --log-file $LOGS_DIRECTORY/%I.log

# 3<>$"${tapdevice}" \
#       --fs tag="homefs,socket=${virtiofs_sock},num_queues=1,queue_size=512" \

[Install]
WantedBy=multi-user.target

And an isolation slice:

sdake@beast-06:/usr/local/bin$ cat sdac.slice
[Unit]
Description=Software Defined Accelerated Compute (SDAC™) Slice
Documentation=man:systemd.special(7)
Before=slices.target

And finally, (overrideable configuration:

sdake@beast-06:/etc/sdacctl.d$ cat 00
A=100
B=100

Would be pleased to work with you on this, or come to a common implementation, or altneratively, learn from each other. As you mention, (bridge networking) is a little challenging. As you see, my bridge networking is incomplete, although I have a complete working example in bash.

One idea I had wanted to approach was using a generator, although it seems too complex. As you can see (if you click through to the github link), there is alot of stuff going on when I launch virtual machines for using VFIO.