ADO Replace Tokens Local

tokenReplace.sh

#!/bin/bash
echo "arguments passed, path: $1, group ids: ${@:2}"

token_replace_path=$1
variable_group_ids=(${@:2})
AZURE_DEVOPS_ORG='{org}'
AZURE_DEVOPS_PROJECT='{project}'

# Initialize an empty JSON object
declare -A json_array
declare -A variable_groups
merged_json="{}"

# Loop through each variable group ID
for group_id in "${variable_group_ids[@]}"; do
    # Fetch variables for the current group
    printf "%-40s" "Fetching variable group id: '$group_id'..."
    #echo -n "Fetching varibale group id: '$group_id' ..."

    response=$(az pipelines variable-group show \
    --group-id "$group_id" \
    --org https://dev.azure.com/$AZURE_DEVOPS_ORG \
    --project $AZURE_DEVOPS_PROJECT \
    --output json)

    group_name=$(echo "$response" | jq -r '.name' )

    echo -n "✅ '$group_name'"
    echo ""

    group_json=$(echo "$response" | jq '.variables')

    json_array[$group_name]=$group_json
    variable_groups[$group_id]=$group_name
done

# loop through json array
for group_id in "${variable_group_ids[@]}"
do
    group_name="${variable_groups[$group_id]}"
    # Merge the fetched variables into the merged_json
    printf "%-40s" "Merging group '$group_name'..."
    merged_json=$(echo "$merged_json" "${json_array[$group_name]}" | jq -s 'add')
    echo -n "✅"
    echo ""
done

# flatten json
printf "%-40s" "Flattening json structure..."
flattened_json=$(echo "$merged_json" | jq 'with_entries(.value |= .value)')
echo -n "✅"
echo ""

# Step 2: Write the flattened JSON to a temporary file for token replacement
TEMP_VARIABLES_FILE=$(mktemp)
echo "$flattened_json" > $TEMP_VARIABLES_FILE

# Use replacetokens CLI to replace tokens within the TEMP_VARIABLES_FILE itself
printf "%-40s" "Variable file token replacement..."
tmp_file_result=$(replacetokens --sources "$TEMP_VARIABLES_FILE" --variables "@$TEMP_VARIABLES_FILE" --token-pattern azpipelines --recursive --log-level off --escape json)
echo -n "✅"
echo -n " $tmp_file_result"
echo ""

# Use the TEMP_VARIABLES_FILE as the variables input on a real token replace
printf "%-40s" "Path: '$token_replace_path' token replacement..."
replacetokens --sources "$token_replace_path" --variables "@$TEMP_VARIABLES_FILE"
echo -n "✅"
echo ""

printf "%-40s" "Temp variable file cleanup..."
rm $TEMP_VARIABLES_FILE
echo -n "✅"

Token Replacement Script Documentation

Introduction

tokenReplace.sh is a Bash script designed to fetch variable groups from Azure DevOps, merge them into a single JSON object, flatten the JSON structure into the correct format expected by replacetokens, and then perform token replacement within specified files. This script is particularly useful in mimicking the behaviour of the ReplaceTokens Azure DevOps task for local testing.

Dependencies

To run this script, you will need:

• Bash shell (typically available on Linux and macOS systems, and Windows via WSL or Git Bash)
• jq - Command-line JSON processor
• az - Azure CLI, for fetching variable groups from Azure DevOps
• replacetokens CLI - For performing token replacement within files

Installing Dependencies

Windows

For Windows users, it's recommended to use Windows Subsystem for Linux (WSL) to run Bash scripts, though it works fine with Git Bash. Once WSL is set up, you can install the dependencies using the Linux instructions below.

Additionally, you can install the Azure CLI and jq using winget:

winget install -e --id Microsoft.AzureCLI
winget install -e --id jqlang.jq

Linux

Using apt (Debian/Ubuntu)

sudo apt update
sudo apt install jq
sudo apt install azure-cli

Using snap

For systems that support snap:

sudo snap install jq
sudo snap install azure-cli --classic

https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux

Installing the az pipelines Extension

After installing the Azure CLI, you can add the az pipelines extension to enable Azure DevOps Pipelines functionality directly from the CLI.

az extension add --name azure-devops

Installing replacetokens

Before installing replacetokens, ensure you have Node.js installed on your system as it is required to use npm (Node Package Manager).

Install Node.js

• On Windows, you can install Node.js using winget:

  winget install -e --id OpenJS.NodeJS

• On Linux, you can use a package manager like apt for Debian/Ubuntu:

  sudo apt update
  sudo apt install nodejs npm

Install replacetokens

Once Node.js is installed, you can install replacetokens globally using npm:

npm i -g @qetza/replacetokens

Azure CLI Authentication

Before running the tokenReplace.sh script, it's essential to authenticate with Azure using the Azure CLI. This step ensures that the script can successfully fetch variable groups from Azure DevOps and perform other Azure-related operations. To authenticate, run the following command:

az login

This command will open a web page where you can log in using your Azure credentials. Once authenticated, the Azure CLI will have the necessary permissions to execute commands related to your Azure resources.

Ensure that your account has the appropriate permissions to access the Azure DevOps variable groups and any other resources the script interacts with

Usage

To use tokenReplace.sh, you will need to pass the path of the file where tokens will be replaced as the first argument, followed by the IDs of the Azure DevOps variable groups as subsequent arguments.

Syntax

./tokenReplace.sh <path-to-file> <variable-group-id-1> <variable-group-id-2> ...

Example

./tokenReplace.sh ./config.json 183 356 428

This command will fetch variables from Azure DevOps variable groups with IDs 183, 356, and 428, merge and flatten them into a single JSON object, and perform token replacement within config.json.

Notes

Permissions

Ensure you have the necessary permissions to access the Azure DevOps variable groups.

Handling Duplicate Properties in JSON Merge

When merging JSON objects from different variable groups, it's possible to encounter duplicate properties. In such cases, the precedence of the merged JSON object's properties is determined by the order in which the variable groups are processed. Specifically, if a property exists in multiple variable groups, the value from the last processed group will take precedence and overwrite any previous values for that property.

This behaviour is crucial to understand for configuring your variable groups and their processing order, ensuring that the intended values are preserved in the final merged JSON object.