Copy a file to a Google Storage Bucket only if it's new or newer

gsutilcpifnewer.sh

#!/bin/bash

# Exists with code 3 if no changes are made

# Be professional
set -euo pipefail

# Check for the correct number of arguments
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <local_file> <bucket_destination>"
    exit 1
fi

# E.g. myfile or /home/me/myfile
local_file=$1
# E.g. gs://mybucket/myfile or gs://mybucket/ (the trailing slash is important to trat it as a directory)
remote_file=$2

# Check if the destination is a directory (ends with a '/')
if [[ "$remote_file" == */ ]]; then
  remote_file="${remote_file}$(basename "$local_file")"
fi

# Calculate the MD5 hash of the local file
local_md5=$(md5sum "$local_file" | cut -d ' ' -f 1)

# Get the MD5 hash of the remote file
remote_md5_b64=$(gsutil stat "$remote_file" 2>/dev/null | grep 'Hash (md5):' | awk '{print $3}' || echo "")

# If the remote file doesn't exist, remote_md5_b64 will be empty
if [ -n "$remote_md5_b64" ]; then
  # Decode the base64-encoded remote MD5 hash and convert it to hex
  remote_md5=$(echo -n "$remote_md5_b64" | base64 --decode | xxd -p | tr -d '\n')
else
  remote_md5="not found"
fi

# Decode the base64-encoded remote MD5 hash and convert it to hex
remote_md5=$(echo -n "$remote_md5_b64" | base64 --decode | xxd -p | tr -d '\n')

# Compare hashes and copy if they don't match
#Debug
#echo "local:  $local_md5"
#echo "remote: $remote_md5"
if [[ "$local_md5" != "$remote_md5" ]]; then
  gsutil cp "$local_file" "$remote_file"
else
  # No new changes
  exit 3
fi

Streamlined usage:

curl -sL -O https://gist.githubusercontent.com/xucian/11629c7759ef3dfc83e00f4357f0dda9/raw/gsutilcpifnewer.sh
chmod +x gsutilcpifnewer.sh && ./gsutilcpifnewer.sh local dest

Note this points to the newest version of the gist. If you want to pin to a specific version for stability, use the full URL (click the Raw button), such as https://gist.githubusercontent.com/xucian/11629c7759ef3dfc83e00f4357f0dda9/raw/<somerevisionhere>/gsutilcpifnewer.sh

Exit code 3 if no changes