Skip to content

Instantly share code, notes, and snippets.

@kcuzner
Created October 31, 2012 20:36
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kcuzner/3989700 to your computer and use it in GitHub Desktop.
Save kcuzner/3989700 to your computer and use it in GitHub Desktop.
Multi-process folder archiving
#!/bin/bash
# purpose: to compress my home folder
#output folder for the files
OUTPUTDIR="/media/My Book/HomeFolderBackup/"
#number of compressing process
NPROC=6
#queue stuff
QUEUE=""
NUM=0
function queue {
QUEUE="$QUEUE $1"
NUM=$(($NUM+1))
}
function regeneratequeue {
OLDREQUEUE=$QUEUE
QUEUE=""
NUM=0
for PID in $OLDREQUEUE
do
if [ -d /proc/$PID ]; then
QUEUE="$QUEUE $PID"
NUM=$((NUM + 1))
fi
done
}
function checkqueue {
OLDCHQUEUE=$QUEUE
for PID in $OLDCHQUEUE
do
if [ ! -d /proc/$PID ]; then
regeneratequeue
break
fi
done
}
#put the list of files into a file
ls -l > archiveList.txt
#and loop through
I=0
cat archiveList.txt | while read line
do
FNAME=$(echo ${line} | awk '{ print $8 }')
tar pcvzf "${OUTPUTDIR}${FNAME}.tar.gz" "${FNAME}" &
PID=$!
queue $PID
while [ $NUM -ge $NPROC ]; do
checkqueue
sleep 0.4
done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment