Backup mongodb database directly to S3

mongodb-backup.sh

# Use environment variable as password (could also be a password stored in a file, read with cat)
# DO NOT use --password argument in mongodump command or authentication will fail
# DO NOT add a file name after --archive argument, or mongodump output won't be sent to stdout
# Use S3 infrequent access storage class to reduce cost
# Use aes-256 encryption of your file on S3 (or your own KMS keys)

# Benefits of this method :
# 1 > You won't see the password sent to mongodump using in ps command (it's safe in a multi-user environment)
# 2 > You don't have to store the backup locally before sending it to S3
# 3 > Everything is done in one line, fits a cronjob easily
# 4 > You can do the "reverse" command and use mongorestore from stdin and awscli to read from S3 !

(echo $BACKUP_PASSWORD) \
  | mongodump --host localhost --port 27017 -u backupUser --authenticationDatabase users --archive --gzip --db mydatabase \
  | aws s3 cp - s3://mybucket/mydatabase/mydatabase-backup-file.agz --storage-class STANDARD_IA --sse --profile s3uploader
  

That's amazing dude... Im trying to get it work over python with subprocess, but i can't...

Any idea how can i make it with python3 !?

I don't see why you couldn't do it with subprocess as long as your command line is correctly build. You could also wrap the whole command in a .sh file and subprocess it ?

HI H-Max, you're right. With subprocess is very easy.

  copy = subprocess.Popen(tos3, stdin=subprocess.PIPE, stdout=None, stderr=None)
  dump = subprocess.run(mongodunmp, stdout=copy.stdin, stderr=subprocess.PIPE)

I made it in this in way, and worked like a charm ! :D

Super awesome!!! My sincere gratitude.

Also, i was wondering how the reverse will work. I have tried to no success:

(echo $BACKUP_PASSWORD) \ | aws s3 cp s3://BucketName/DatabaseBackUp.agz --storage-class STANDARD_IA --sse --profile admin \ | mongorestore --gzip --archive=DatabaseBackUp.agz --nsInclude "FilterWord.*"

Perhaps i am not getting this correct. I am new at it.

It's a bit more complicated and I can't test it on my side now (not working on this anymore)

The order of operations is different (you need to send both the password and the stdin to the mongorestore) and I don't remember exactly how to do it.

Maybe try something like that:

(echo $BACKUP_PASSWORD) \
    | mongorestore --gzip --archive --nsInclude "FilterWord.*" \
    < aws s3 cp s3://BucketName/DatabaseBackUp.agz - --profile admin

(the single dash in the s3 command is important)

Thank you so much for the prompt response. I will give it a go.
Regards.

Hi @bolanlesd..

You are not handling right stdin and stdout ..

Here, H-Max is sending the dump to stdout and redirecting it to stdin to aws s3 cp for sending to S3, which is awesome 'cuase you are not working with the local FS.
You need to do the same, at reverse, if you want .. If not, you need to download first and the restore..

# According to doc for 'aws s3 cp' 
# cp 
#  <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>

aws s3 cp s3://BucketName/DatabaseBackUp.agz . --storage-class STANDARD_IA --sse --profile admin \ 
| (echo $BACKUP_PASSWORD) \
| mongorestore --gzip --archive=DatabaseBackUp.agz --nsInclude "FilterWord.*"

Check if this works for you ..

Thank you for the contribution @luqpy, I am currently looking into your suggestion as well, i am trying to make it work as I hope for success.

I appreciate your help.

So this worked for me for the restore process:
But it will need to be run from the Primary DB instance only (i tried on a secondary instance but it will not work with error Failed: EOF)

#Download backup from S3
aws s3 cp s3://BucketName/$DBNAME.agz /tmp

(echo $BACKUP_PASSWORD) | mongorestore --host $HOST --port 27017 --authenticationDatabase admin -u admin --gzip --archive=/tmp/$DBNAME.agz --nsInclude "$DBNAME.*"

Also, it will need to be run twice if the database does not exist before. Don't know if there could be a walk around so it just runs once.

Thank you so much for the help.

Does not work:

Failed: archive writer: error writing data for collection `blah.blah` to disk: short write / write /dev/stdout: broken pipe

same error as @talaikis anyoone could help

Hi @mehdibenfeguir, could you give more information ??

Script? Full error output? Environment?

Maybe is some kind of error with the Mongodump or mongoDB versions.. Any additional info would be great to help you!

Regards :)