Skip to content

Instantly share code, notes, and snippets.

@DavidWittman
Created December 23, 2015 16:47
Show Gist options
  • Save DavidWittman/0edb563f1bda9bb89174 to your computer and use it in GitHub Desktop.
Save DavidWittman/0edb563f1bda9bb89174 to your computer and use it in GitHub Desktop.
Script to merge .part files from Amazon S3
#!/usr/bin/env bash
if [[ $# -ne 1 ]]; then
echo "Merge matching *.part files in a directory"
echo
echo "usage: $0 <directory>"
exit 1
fi
DIRECTORY="$1"
FILES=$(find "${DIRECTORY}" -regextype sed -regex '.*part[[:digit:]]\+$' | sed 's/\.part[[:digit:]]\+$//' | uniq)
while read -r BASE; do
COUNT="$(\ls -l "${BASE}".part* | wc -l)"
MAX="$((COUNT-1))"
echo "Merging ${COUNT} parts with base ${BASE}"
# We know there are COUNT parts which belong to this BASE
# Use `seq` to create an array of consecutive filenames
# to preserve ordering and prevent whitespace issues.
IFS=$'\n' PARTS=($(seq -f "${BASE}.part%g" 0 ${MAX}))
stat "${PARTS[@]}" >/dev/null 2>&1
if [[ $? -ne 0 ]]; then
echo "Error: Missing part for file ${BASE}" 1>&2
exit 1
fi
cat "${PARTS[@]}" > "${BASE}"
done <<< "$FILES"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment