Turing Pi 2: On BMC host flash conveniently multiple CM4

README.md

init-nodes.sh

The init-nodes.sh script is a utility for the Turing Pi 2 to initialize and configure multiple CM4 nodes in a structured and automated manner. It operates by looping through specified node numbers, performing actions like copying configurations, power cycling, and more.

Prerequisites

• Tested with BMC version 1.1.0 -- bug reports with older versions are welcome (see below).
• The script requires sh.
• It should be run with root permissions.
• The image file to be flashed should be provided.
• To flash DietPi
  • mkdir -p /mnt/sdcard/dietpi/tpl and place https://github.com/MichaIng/DietPi/blob/master/dietpi.txt in there
  • put DietPi_RPi-ARMv8-Bookworm.img in /mnt/sdcard/images
• To flash Raspberry Pi OS
  • put e.g. 2023-05-03-raspios-bullseye-armhf-lite.img in /mnt/sdcard/images
  • mkdir -p /mnt/sdcard/raspios/tpl and place your file userconf (without extension) in there. (Optional, default: Creates user pi with password raspberry.)

Usage

# ./init-nodes.sh [NUM_NODES] [HOSTNAME_PREFIX] [IMAGE_FILE]
# curl -k https://gist.githubusercontent.com/cprima/e09a5d2f4151f38cc98f2b2a196a0ee1/raw/c6f02c5291c7e127d763429e32aab84f6282ea99/init-nodes.sh  | UNATTENDED_RUN=1 sh -s -- 3 tpi #(get the latest URL by clicking "raw" for init-nodes.sh on this Gist)
# #For a single integer input for NUM_NODES:
# ./init-nodes.sh 3 tpi-node your_image_file.img
# #For a range input for NUM_NODES:
# ./init-nodes.sh 2..4 tpi-node your_image_file.img
# #For a Raspberry Pi OS installation:
# UNATTENDED_RUN=1 ./init-nodes.sh 1..3 tpi 2023-05-03-raspios-bullseye-armhf-lite.img

Arguments:

• NUM_NODES: A parameter to define nodes for the script's operation. (Mandatory, up to 4 nodes).

  • If a single integer, such as 3, is specified, the script will run actions on nodes from 1 through to NUM_NODES.
  • When given a range (like 2..4), the script begins at the first specified node and ends with the last one in the range.

• HOSTNAME_PREFIX: The prefix for the hostname. A zero-padded node number will be appended to this prefix to form the full hostname (Optional, default: node).

• IMAGE_FILE: Name of the image file you want to flash. Provide just the filename. The script expects it to be located in /mnt/sdcard/images (Optional).

Environment Variables:

• UNATTENDED_RUN: If set, the script will not prompt for user confirmation during the loop and will directly execute the necessary steps.

• START_POSITION: Specifies the starting position in the loop (Default: 1).

Steps

1. Preparation:

   • Check if the image file exists.
   • If the filename contains "dietpi", the script checks for the existence of /mnt/sdcard/dietpi/tpl/dietpi.txt. This is the original dietpi.txt as it is contained in the image.
   • If the filename contains "raspios", the script checks for the existence of /mnt/sdcard/raspios/tpl/userconf. If it exists it is copied to the boot partition. If not, an equivalent of the username pi and password raspberry is generated.

2. Node Loop:

   • For each node, the script performs several operations:
     • Copy specific configuration files.
     • Modify configurations as required.
     • Power cycle nodes.
     • And more.

3. Completion:

   • At the end of the script, a progress summary is provided, detailing the actions performed on each node.

Notes

• The script keeps track of milestones during its execution. In case of any issues or interruptions, the progress summary can provide insights into the steps completed.

Reporting Issues

If you encounter any issues or have suggestions for improvement, please report them as a comment to the gist.

License

This project is licensed under the MIT License.

Author

Christian Prior-Mamulyan

Source

The script can be found at this gist.

init-nodes.sh

#!/bin/sh

: '
init-nodes.sh

Usage:
  ./init-nodes.sh [NUM_NODES] [HOSTNAME_PREFIX] [IMAGE_FILE]

Arguments:
  NUM_NODES:          The number of nodes you want the script to loop through (maximum of 4).
                      Default: 3.
                      - If a single integer, such as 3, is specified, the script will run actions on nodes from 1 through to NUM_NODES.
                      - When given a range (like 2..4), the script begins at the first specified node and ends with the last one in the range.

  HOSTNAME_PREFIX:    The prefix for the hostname. A zero-padded node number will be appended to this prefix to form the full hostname.
                      Default: "node".
  IMAGE_FILE:         The filename of the image to be flashed.
                      Default: DietPi_RPi-ARMv8-Bookworm.img.

Examples:
  ./init-nodes.sh 4 node MyCustomImage.img
  ./init-nodes.sh 2..3 mynode
  ./init-nodes.sh

Environment Variables:
  START_POSITION:     Sets the starting point for the node sequence.
                      Example: START_POSITION=2 ./init-nodes.sh 4  -> This will loop over nodes 2 to 4.
                      Deprecated, use the range notation x..y instead.
  UNATTENDED_RUN:     If set, the script will run without prompting the user for confirmation to reboot during each node iteration.
                      Example: UNATTENDED_RUN=1 ./init-nodes.sh
'

# Ensure the script is being run as root
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root!"
    exit 1
fi

HOSTNAME_PREFIX="${2:-node}"
IMAGE_FILE="${3:-DietPi_RPi-ARMv8-Bookworm.img}"
IMAGE_PATH="/mnt/sdcard/images/$IMAGE_FILE"
TPI_SETTLE_DELAY=12

# Check if the image file exists
if [ ! -f "$IMAGE_PATH" ]; then
    echo "Error: Image file '$IMAGE_PATH' does not exist."
    exit 1
fi

# Check for DietPi-specific operations
LOWER_CASE_FILENAME=$(echo "$IMAGE_FILE" | tr '[:upper:]' '[:lower:]')
IS_DIETPI=false
IS_RASPIOS=false
if echo "$LOWER_CASE_FILENAME" | grep -q "dietpi"; then
    IS_DIETPI=true
    if [ ! -f "/mnt/sdcard/dietpi/tpl/dietpi.txt" ]; then
        echo "Error: DietPi detected but '/mnt/sdcard/dietpi/tpl/dietpi.txt' file does not exist."
        exit 1
    fi
fi
if echo "$LOWER_CASE_FILENAME" | grep -q "raspios"; then
    IS_RASPIOS=true
fi

# Start position for node sequence
START_POSITION="${START_POSITION:-1}"

# Initialize a string to accumulate progress notes for milestones reached during the script's execution
progress_notes=""

# Initialize a string to keep track of successful nodes
successful_nodes=""

display_progress_summary() {
    # Print the accumulated progress notes for a summary of all the milestones reached during script execution
    # echo -e does not work in sh
    printf "%s\n" "Progress summary:${progress_notes}"
}

# Function to perform power off and on operations for each successful node
# unused
power_cycle_successful_nodes() {
    # For each successful node, perform the power off and power on operations
    for successful_node in $successful_nodes; do
        # Power off the node
        tpi -p off -n "$successful_node"
        sleep $TPI_SETTLE_DELAY
        # Power on the node
        tpi -p on -n "$successful_node"
    done
}

# exit trap
teardown() {
    # power_cycle_successful_nodes
    # echo "Successful: $successful_nodes"
    display_progress_summary
}

trap teardown EXIT

# Formats a given node number into a zero-padded two-digit string.
# Usage: formatted_node=$(format_node_number "$node")
format_node_number() {
    node_num="$1"
    printf "%02d" "$node_num"
}

#region parse_range (this comment is for folding in VSCode)
parse_range() {
    : '
    parse_range

    Description:
    Parses a numeric input to determine the start position and number of nodes. The input can be
    a single number or a numeric range in the format of "X..Y".

    Parameters:
    - input (string): A representation of a single number or a numeric range.

    Expected Input Formats:
    - Single number between 1 and 4 (inclusive), e.g., "2"
    - Numeric range between 1 and 4 in the format "X..Y", e.g., "1..3"

    Outputs:
    - For a single number: "START_POSITION=1 NUM_NODES=<input>"
    - For a numeric range: "START_POSITION=<X> NUM_NODES=<Y>"

    Constraints:
    - Both X and Y in the range format must be between 1 and 4 (inclusive).
    - X must not be greater than Y in the range format.

    Returns:
    Echoes the determined range or an error message for invalid inputs.
    Returns 0 on success and 1 on error.
    '
    input="$1"

    # Check for a single integer input
    case "$input" in
    [1-4])
        #echo "START_POSITION=1 NUM_NODES=$input"
        START_POSITION=1
        NUM_NODES=$input
        return 0
        ;;
    [1-4]..[1-4])
        # Split the input on the '..' delimiter
        start="${input%%..*}"
        end="${input##*..}"

        # Ensure start is not greater than end
        if [ "$start" -le "$end" ]; then
            #echo "START_POSITION=$start NUM_NODES=$end"
            START_POSITION=$start
            NUM_NODES=$end
            return 0
        else
            echo "Error: Start value is greater than end value."
            return 1
        fi
        ;;
    *)
        echo "Error: Invalid input."
        return 1
        ;;
    esac
}

# # Test
# for i in 1 2 3 4 "2..4" "3..4" "1..3" "5" "5..2" "2..5"; do
#     parse_range "$i"
#     if [ $? -eq 0 ]; then
#         echo "Test passed!"
#     else
#         echo "Test failed!"
#     fi
#     echo "............"
# done

#endregion parse_range

NUM_NODES="${1:-3}"

parse_range "${NUM_NODES}"
if [ $? -eq 0 ]; then
    :
else
    exit 1
fi
[ "$NUM_NODES" -le 4 ] || {
    echo "Maximum allowed nodes is 4."
    exit 1
}

#region get_os_version
get_os_version() {
    while IFS="=" read -r key value; do
        if [ "$key" = "VERSION" ]; then
            # Removing quotes if present
            value="${value%\"}"
            value="${value#\"}"
            echo "$value"
            return
        fi
    done </etc/os-release
}

# # Usage:
# version=$(extract_version_from_bmc_api)
# if [ $? -eq 0 ]; then
#     echo "Extracted version: $version"
# else
#     echo "Failed to extract version."
# fi

#endregion

BMC_VERSION=$(get_os_version)

echo "################################################################################"
echo "Configuration Summary:"
echo "--------------------------------------------------------------------------------"
echo "BMC version: $BMC_VERSION"
echo "First node: $START_POSITION"
echo "Last node: $NUM_NODES"
echo "Hostname Prefix: $HOSTNAME_PREFIX"
echo "Image path and file: $IMAGE_PATH"
echo "Unattended Run: $UNATTENDED_RUN"
# if [ -f "/mnt/sdcard/dietpi/tpl/Automation_Custom_Script.sh" ]; then
# echo "DietPi Automation_Custom_Script.sh: Exists"
# fi
echo "================================================================================"

# Loop to turn off each node
for node in $(seq "$START_POSITION" "$NUM_NODES"); do
    formatted_node=$(format_node_number "$node")
    echo "Turning off node ${HOSTNAME_PREFIX}${formatted_node}..."
    tpi -p off -n "$node" >/dev/null 2>&1
    sleep "$TPI_SETTLE_DELAY" # Letting components settle
done

# Main loop over the number of nodes
for node in $(seq "$START_POSITION" "$NUM_NODES"); do
    formatted_node=$(format_node_number "$node")
    echo "********************************************************************************"
    echo "Current node: $node"
    echo "--------------------------------------------------------------------------------"

    # Flash the image file
    tpi -n "$node" -l -f "$IMAGE_PATH"

    # Load the node as a mass storage device
    tpi -n "$node" -m && echo ""

    # Identify the storage device that was found
    storage_device=$(dmesg | tail -12 | grep -o "sd[a-z]" | tail -1)

    # Mount the identified device
    if [ -n "$storage_device" ]; then

        # Mount the identified storage device
        mount "/dev/${storage_device}1" /mnt/bootfs

        # region DietPi-specific operations
        if $IS_DIETPI; then
            output_dir="/mnt/sdcard/dietpi/${HOSTNAME_PREFIX}${formatted_node}"

            # Check if the output directory exists, if not create it
            [ ! -d "$output_dir" ] && mkdir -p "$output_dir"

            # Use sed to replace the hostname in the template file and output to the node-specific directory
            sed "s/AUTO_SETUP_NET_HOSTNAME=.*$/AUTO_SETUP_NET_HOSTNAME=${HOSTNAME_PREFIX}${formatted_node}/" "/mnt/sdcard/dietpi/tpl/dietpi.txt" >"${output_dir}/dietpi.txt"
            # Copy the node-specific file to /mnt/bootfs
            cp "${output_dir}/dietpi.txt" /mnt/bootfs/dietpi.txt

            # Copy cmdline.txt to /mnt/bootfs
            cp "/mnt/sdcard/dietpi/tpl/cmdline.txt" /mnt/bootfs/cmdline.txt

            # Update config.txt for UART configuration
            if grep -q "^enable_uart=" /mnt/bootfs/config.txt; then
                sed -i "s/^enable_uart=.*/enable_uart=1/" /mnt/bootfs/config.txt
            else
                # echo "\n# Enable UART" >>/mnt/bootfs/config.txt
                printf "\n# Enable UART" >>/mnt/bootfs/config.txt
                echo "enable_uart=1" >>/mnt/bootfs/config.txt
            fi

            # Check for the existence of Automation_Custom_Script.sh
            if [ -f "/mnt/sdcard/dietpi/tpl/Automation_Custom_Script.sh" ]; then
                # depends on AUTO_SETUP_CUSTOM_SCRIPT_EXEC=0 (see dietpi.txt)
                cp "/mnt/sdcard/dietpi/tpl/Automation_Custom_Script.sh" "/mnt/bootfs/"
            fi
        fi
        #endregion

        # region Raspberry Pi OS-specific operations
        if $IS_RASPIOS; then
            # Check if /mnt/sdcard/raspios/tpl/userconf exists
            if [ -f "/mnt/sdcard/raspios/tpl/userconf" ]; then
                # Copy it to /mnt/bootfs
                cp "/mnt/sdcard/raspios/tpl/userconf" "/mnt/bootfs/"
            else
                # If it doesn't exist, echo the specified string into /mnt/bootfs/userconf
                # shellcheck disable=SC2016
                echo 'pi:$6$c70VpvPsVNCG0YR5$l5vWWLsLko9Kj65gcQ8qvMkuOoRkEagI90qi3F/Y7rm8eNYZHW8CY6BOIKwMH7a3YYzZYL90zf304cAHLFaZE0' >"/mnt/bootfs/userconf"
            fi
            # Touch the file /mnt/bootfs/ssh to create it
            touch "/mnt/bootfs/ssh"
            # Check for the last occurrence of the line '[all]' in /mnt/bootfs/config and add 'enable_uart=1' after it
            last_line=$(awk '/\[all\]/ {line=NR} END {print line}' "/mnt/bootfs/config.txt")

            awk -v line="$last_line" 'NR==line+1 {print "enable_uart=1"} 1' "/mnt/bootfs/config.txt" >"/mnt/bootfs/config.tmp" && mv "/mnt/bootfs/config.tmp" "/mnt/bootfs/config.txt"

            [ ! -d /mnt/rootfs ] && mkdir -p /mnt/rootfs
            mount "/dev/${storage_device}2" /mnt/rootfs
            echo "${HOSTNAME_PREFIX}${formatted_node}" >/mnt/rootfs/etc/hostname
            umount /mnt/rootfs
        fi
        #endregion

        # Unmount the storage device
        umount /mnt/bootfs
    else
        # Add a note to the progress indicating failure to identify the device
        progress_notes="$progress_notes
Node $node: Failed to identify the device for node $node from dmesg."
        continue
    fi

    # Append the current node to the list of successful nodes
    successful_nodes="$successful_nodes $node  (${HOSTNAME_PREFIX}${formatted_node})"
    echo "--------------------------------------------------------------------------------"
    echo "Successful nodes: $successful_nodes"

    # Prompt the user for action
    if [ -z "$UNATTENDED_RUN" ]; then
        # echo -e "\n\nTrigger the restart of the node? [y/N]: "
        # echo -e does not work in sh
        printf "\n\nTrigger the restart of the node? [y/N]: "
        read -r response
        if [ "$response" != "y" ]; then
            progress_notes="$progress_notes
Node $node: User aborted after image flash."
            continue
        fi
    fi

    # Finalize node setup
    tpi -n "$node" -x >/dev/null 2>&1
    sleep $TPI_SETTLE_DELAY
    tpi -u host -n "$node" >/dev/null 2>&1
    sleep $TPI_SETTLE_DELAY
    tpi -p off -n "$node" >/dev/null 2>&1
    sleep $TPI_SETTLE_DELAY
    tpi -p on -n "$node" >/dev/null 2>&1
    echo "================================================================================"
    echo ""

    # Add a note to the progress indicating success
    progress_notes="$progress_notes
Node $node: Completed setup."

done

# @see exit trap

Roadmap:

[ ] Get feedback
[X] rework the parameter indicating the node, from an environment variable for the STARTING_POSITION and the number of nodes to start..end notation
[X] add checks for a Raspbian installation
[ ] publish screenrecording of the script running
[ ] probe for CM4 (if that makes sense, needs investigation)
[ ] re-mount /mnt/sdcard if not mounted (but e.g. just inserted)
[X] explore "gist as a repository"

Released v3 which detects a RaspiOS image and does a headless install (including a userconf with pi:raspberry as username and password).

Which BMC version(s) is this compatible with?

Which BMC version(s) is this compatible with?

I tested it with 1.1.0
Added this to the README.md

In case there is an incompatibility with previous versions then I could extend BMC_VERSION=$(get_os_version) with a check.

Feel free to let me know if the documentation is lacking. Such script are only helpful if usage is clear.
I surely have blind spots.