Skip to content

Instantly share code, notes, and snippets.

@labbots
Last active December 22, 2016 16:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save labbots/d695b741bd59025fedcc3817203ae93d to your computer and use it in GitHub Desktop.
Save labbots/d695b741bd59025fedcc3817203ae93d to your computer and use it in GitHub Desktop.
Simple test to check the usage of dd command. This script allows to split a file into chunks and merge them back from chunks to original file using dd command in linux. https://labbots.com/split-and-merge-file-using-dd-command/
#!/bin/bash
#!/bin/bashset -e
PROGNAME=${0##*/}
SHORTOPTS="mbs:d:p:"
LONGOPTS="merge,split,source:,destination:,prefix:"
set -o errexit -o noclobber -o pipefail
OPTS=$(getopt -s bash --options $SHORTOPTS --longoptions $LONGOPTS --name $PROGNAME -- "$@" )
eval set -- "$OPTS"
#Base variable declaration
FILE=""
DESTINATION=""
MERGE=false
SPLIT=false
PREFIX="split"
BLOCK_SIZE=10000
while true; do
case "$1" in
-m | --merge ) MERGE=true; shift ;;
-b | --split ) SPLIT=true; shift ;;
-s | --source ) FILE="$2"; shift 2 ;;
-d | --destination ) DESTINATION="$2"; shift 2 ;;
-p | --prefix ) PREFIX="$2"; shift 2 ;;
-- ) shift; break ;;
* ) break ;;
esac
done
# if both merge and split flags are set then throw error
if [ "$MERGE" = true ] && [ "$SPLIT" = true ]; then
echo "Cannot set flag merge and split at the same time."
exit 1
fi
# if no flags are set then throw error
if [ "$MERGE" = false ] && [ "$SPLIT" = false ]; then
echo "Atlest one of the merge or split flag must be set."
exit 1
fi
# If Destination folder is specified then we check whether the folder is there, if not the folder is created.
if [ ! -z "$DESTINATION" ] && [ ! -d "$DESTINATION" ];then
mkdir -p $DESTINATION
if [ $? -ne 0 ];then
echo "Creating destination directory failed."
exit 1
fi
DESTINATION="$(readlink -f $DESTINATION)"
fi
if [ ! -z "$DESTINATION" ] && [ -d "$DESTINATION" ];then
DESTINATION="$(readlink -f $DESTINATION)"
fi
# If Destination is empty then create files in the same directory as of source file
if [ -z "$DESTINATION" ];then
DESTINATION="$(dirname "$(readlink -f $FILE)")"
fi
if [ ! -z "$1" ];then
FILE="$1"
fi
if [ ! -z "$SPLIT" ] && [ "$SPLIT" = true ];then
if [ ! -f "$FILE" ];then
echo "Invalid file name : $FILE"
exit 1
fi
FULLFILENAME=$(basename $FILE)
FILENAME="${FULLFILENAME%.*}"
FILESIZE=$(stat -c%s "$FILE")
BLOCKS=$(((FILESIZE + (BLOCK_SIZE + 1)) / BLOCK_SIZE))
echo "$FILESIZE"
echo "$BLOCKS"
for (( block = 0; block < $BLOCKS; block += 1 ))
do
echo $block;
START_RANGE=$((block*BLOCK_SIZE))
END_RANGE=$((((block +1) * BLOCK_SIZE) - 1))
if [ "$block" -eq $((BLOCKS-1)) ]
then
END_RANGE=$FILESIZE
fi
echo "$START_RANGE : $END_RANGE"
dd if=$FILE skip=$block bs=$BLOCK_SIZE count=1 status=noxfer 2> /dev/null > "${DESTINATION}/${PREFIX}|${FILENAME}|${block}"
done
fi
if [ ! -z "$MERGE" ] && [ "$MERGE" = true ];then
if [ ! -d "$FILE" ];then
echo "Invalid source folder : $FILE"
exit 1
fi
FILENAME=$(find -name "split|*" | sort -V | head -1 | sed -n "s/.*split|\([a-zA-Z0-9_\-]*\)|[0-9]*/\1/p")
#cat $(find $FILE -name "${PREFIX}|*" | sort -V) > "${DESTINATION}/${FILENAME}"
FILES=$(find -name "split|*" | sort -V | sed ':a;N;$!ba;s/\n/ /g')
for f in $FILES
do
dd if=$f of=${DESTINATION}/${FILENAME} conv=notrunc oflag=append bs=$BLOCK_SIZE
done
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment