Dynamically isolate CPU cores for each virtual machine without any conflicts using systemd

libvirt-isolate-cores.sh

#!/bin/bash
if [ $(id -u) -ne 0 ]; then
	echo "You must run this as root."
	exit;
fi

IS_VERBOSE="0"

# Create temporary file to manage active core isolations (for all virtual machines)
ISOLATED_CPU_FILE="/tmp/libvirt-isolated-cpus.txt"
[ ! -f "$ISOLATED_CPU_FILE" ] && touch "$ISOLATED_CPU_FILE"

# Handle args
while [ $# -gt 0 ]; do 
     case $1 in 
	  -l | --list)
	       # add ability to list a specific virtual machine's cores later
	       cat "$ISOLATED_CPU_FILE"
	       exit; 
	  ;;
	  -n | --name)
	       VIRTUAL_MACHINE_NAME="$2"
	  ;;
	  -c | --cores)
	       ISOLATE_THESE_CORES="$2"
	  ;;
	  -a | --add)
	       IS_ADDING="1"
	  ;;
	  -r | --remove)
	       IS_ADDING="0"
	  ;;
	  -v | --verbose)
	       IS_VERBOSE="1"
	  ;;
     esac
     shift
done

if [ -z "$VIRTUAL_MACHINE_NAME" ]; then    
     echo "You need to pass a --name"    
     exit     
fi    

# Add this virtual machine's entry to the file if it doesn't exist:
if ! cat "$ISOLATED_CPU_FILE" | grep -q "^$VIRTUAL_MACHINE_NAME"; then 
     echo "$VIRTUAL_MACHINE_NAME" >> "$ISOLATED_CPU_FILE"
     [ "$IS_VERBOSE" -eq 1 ] && echo "Added new entry for $VIRTUAL_MACHINE_NAME" 
fi

# Sanitize --cores 
range=$(echo "$ISOLATE_THESE_CORES" | grep -o "\b\([0-9]*-[0-9]*\)\b")

if [ $(echo "$range" | wc -c) -gt 1 ]; then # They're adding cores with a range 
     range_min=$(echo "$range" | cut -d "-" -f1)
     range_max=$(echo "$range" | cut -d "-" -f2)  

     [ "$IS_VERBOSE" -eq 1 ] && echo "Processing core range $range_min to $range_max" 

     for ((i = $range_min; i <= $range_max; i++)); do
	  sanitized_core_list="$sanitized_core_list $i"	  
     done
else # They're adding a separated list
     cores=$(echo "$ISOLATE_THESE_CORES" | grep -o "[0-9]*") 

     for i in $cores; do
	  sanitized_core_list="$sanitized_core_list $i"
     done
fi

[ "$IS_VERBOSE" -eq 1 ] && echo "Processed core list: $sanitized_core_list" 
[ -z "$sanitized_core_list" ] && exit; 

if [ "$IS_ADDING" -ge 1 ]; then 			
     sed -i "/^$VIRTUAL_MACHINE_NAME/c $VIRTUAL_MACHINE_NAME $sanitized_core_list" "$ISOLATED_CPU_FILE" 
else
     sed -i "/^$VIRTUAL_MACHINE_NAME/ s/$sanitized_core_list//" "$ISOLATED_CPU_FILE"
fi

if [ "$IS_VERBOSE" -eq 1 ]; then 
     if [ "$IS_ADDING" -eq 1 ]; then 
	  echo "Adding isolated cores for $VIRTUAL_MACHINE_NAME"
     else 
	  echo "Removing isolated cores from $VIRTUAL_MACHINE_NAME"
     fi
fi

# Finally, read from the file containing all isolated cores, relay it to Systemd
all_isolated_cores=$(cat "$ISOLATED_CPU_FILE")
allowed_cores=""

for ((i = 0; i <= $(lscpu -p --all | grep "^[0-9]" | wc -l) - 1; i++)); do 
     if ! echo "$all_isolated_cores" | grep -q "\s$i\b"; then 
	  allowed_cores="$allowed_cores$i,"
     fi 
done

[ -z "$allowed_cores" ] && exit;
allowed_cores="${allowed_cores::-1}"

[ "$IS_VERBOSE" -eq 1 ] && echo "Cores allowed on host: $allowed_cores" 
systemctl set-property --runtime -- system.slice AllowedCPUs=$allowed_cores
systemctl set-property --runtime -- user.slice AllowedCPUs=$allowed_cores
systemctl set-property --runtime -- init.scope AllowedCPUs=$allowed_cores

Dynamic Libvirt CPU Isolation

Scenario: You need to isolate the cores at runtime for several libvirt virtual machines that may use different cores, but you don't want to conflict with existing isolated cores and you want the cores made available to the host again as soon as the VMs using them have powered off.

1. Add isolate-cores.sh

wget "https://gist.githubusercontent.com/Ethorbit/8cf475e2b4deb3d22e4aa7c5f1703cd0/raw/f06a5ac78787d6164f0fd9a60245f444259bc6ca/libvirt-isolate-cores.sh" -O /usr/bin/isolate-cores.sh && chmod +x /usr/bin/isolate-cores.sh

2. Create the hook files for your virtual machines

Libvirt executes scripts in directories that have the same name as a VM when it starts, stops, etc

Create these two files:

• /etc/libvirt/hooks/qemu.d/your vm's name/prepare/begin/start.sh
• /etc/libvirt/hooks/qemu.d/your vm's name/release/end/revert.sh

(Make sure to chmod +x them)

Tip: you can create a symbolic link pointing to a single directory instead if many virtual machines need the same hooks (e.g.,
ln -s /etc/libvirt/hooks/qemu.d/FourCores /etc/libvirt/hooks/qemu.d/your vm's name)

3. Edit the hook files

Inside the prepare script, add:

GUEST_NAME="$1"
isolate-cores.sh --name "$GUEST_NAME" --cores "0,1,2,3" --add

Inside the release script, add:

GUEST_NAME="$1"
isolate-cores.sh --name "$GUEST_NAME" --cores "0,1,2,3" --remove

Change the values to the cores you want to isolate from the host.

Tip: You can also do a range (e.g., "0-3") instead.

4. Restart libvirt

systemctl restart libvirtd

5. Test it

Start the VM and see if the entry was added.

isolate-cores.sh --list

You should see something like this when it's on:

My-VM-Name 0 1 2 3

And when turned off:

My-VM-Name