MongoDB backup on readonly slaves

gistfile1.txt

You can't simply run a backup on a specific node in a cluster, since its role can change to master and lead to an overload for it.
You need some intelligence to choose the right node (slave) and allow multiple nodes to execute the backup; you want to be safe without
designing a specific node to run it.

This is my approach:
* schedule the script below in any MongoDB host
* the script will exit if it's running on a master
* if it's a slave, it will wait a random interval, check if there's a lock document on the master and if not present perform the backup.

in such a way you are covered from any single node failure

#!/bin/bash
#set -x # for debugging
DBUSERNAME=''
DBPASSWORD=''

# Please specify a DB!
DB=$1
#check if it's running on the master
MASTER=$(mongo -u ${DBUSERNAME} -p ${DBPASSWORD} --quiet --eval 'rs.isMaster().ismaster' admin)
if [ "$MASTER" == 'true' ];then
   echo running on master quitting
   exit
else
   SLEEP=$(shuf -i 0-5 -n 1)
   echo running on a slave, sleeping for $SLEEP seconds
   sleep $SLEEP
   LOCK=$(echo -e "use $DB\n rs.slaveOk()\n db.lock.find({\"status\":\"bck created\"})\n" |mongo -u ${DBUSERNAME} -p ${DBPASSWORD} --quiet admin)
   if [ "$LOCK" == "switched to db $DB" ];then
	echo "backing up!"
	# find master ip
	IP=$(mongo -u ${DBUSERNAME} -p ${DBPASSWORD} --quiet --eval 'db.isMaster().primary' admin)
	echo informing master $IP, this node will hold the lock
	# all slaves are read-only, so we connect to master to acquire the lock :) 
	echo -e "use $DB\n db.lock.insert( {\"createdAt\": new Date(),\"host\": hostname(),\"status\":\"bck created\"})\n" | mongo -u ${DBUSERNAME} -p ${DBPASSWORD} --host=${IP} --quiet admin 
  	
  #decide a directory to save
	mongodump -u ${DBUSERNAME} -p ${DBPASSWORD} -d ${DB} --authenticationDatabase=admin 

	# connect to master to remove the lock :)
	echo -e "use $DB\n db.lock.remove({\"status\":\"bck created\"})" | mongo -u ${DBUSERNAME} -p ${DBPASSWORD} --host=${IP} --quiet admin 
   else
        echo "another node is backing up:"
	echo $LOCK
   fi

fi