Skip to content

Instantly share code, notes, and snippets.

@Hiestaa
Last active June 21, 2017 20:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Hiestaa/ad7160c6fbed32d445499bdec2407bb4 to your computer and use it in GitHub Desktop.
Save Hiestaa/ad7160c6fbed32d445499bdec2407bb4 to your computer and use it in GitHub Desktop.
mongorestore too many open files error workaround with per collection restore
#!/bin/bash
if [ $# -eq 0 ]; then
echo "usage: `basename $0` pathToDb [dbName]"
exit 1
fi
pathToDb=$1
if [ $# -eq 1 ]; then
# TODO: use only the depth-most folder name as database name when not provided
database=$pathToDb
else
database=$2
fi
echo "# FYI: system limits"
echo "> ulimit -a"
ulimit -a
# A file named `.imported-collections` will be created.
# Use the command `cat .imported_collections | xargs rm` before rerunning the script
# it an error occurs
# echo "# Dropping database..."
# echo "> mongo $database --eval \"db.dropDatabase();\""
# mongo $database --eval "db.dropDatabase();"
for file in $( ls "$pathToDb/" | grep -iE \.bson$ | grep -v system ); do
collection=$( echo $file | cut -d'.' -f1 )
echo "> mongorestore --noIndexRestore --db $database --collection \"$collection\" \"$pathToDb/$file\""
mongorestore --noIndexRestore --db $database --collection "$collection" "$pathToDb/$file"
exitCode=$?
if [ $exitCode -ne 0 ]; then
# the hard way - restart the server and re-attempt importing this collection
# Only valid for MacOS X, mongodb 3.2 installed with homebrew.
echo "# ERROR: process excited with code: $exitCode"
if [ -e "$pathToDb/$file" ]; then
echo "# Restarting mongo..."
echo "> brew services restart mongodb@3.2"
brew services restart mongodb@3.2
echo "# Waiting a bit for server to be back up..."
echo "> sleep 10"
sleep 10
echo "# Erasing collection data"
echo "> mongo $database --eval \"db['$collection'].remove({});\""
mongo $database --eval "db['$collection'].remove({});"
echo "# Retrying collection import..."
echo "> mongorestore --noIndexRestore --db $database --collection \"$collection\" \"$pathToDb/$file\""
mongorestore --noIndexRestore --db $database --collection "$collection" "$pathToDb/$file"
echo "$pathToDb/$file" >> .imported-collections
else
echo "# File $pathToDb/$file doesn't seem to exist."
fi
else
echo "$pathToDb/$file" >> .imported-collections
fi
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment