Run gitlab backup script, upload repo and folders to s3, log and inform via sendgrid email

gitlab-backup.sh

LOGFILE="/var/log/gitlab-backup.log"

echo "**************START*************************" >> $LOGFILE

#---------1. GITLAB git repos backup ---------
echo "Start the gitlab backup process" >> $LOGFILE

/opt/gitlab/bin/gitlab-rake gitlab:backup:create SKIP=uploads >> $LOGFILE
echo "git backup done" >> $LOGFILE
echo "Now, uploading gitlab repo to s3" >> $LOGFILE

DATE=`date +%Y_%m_%d`
FILENAME="*${DATE}*gitlab_backup*"
EE="$(find /mnt/data/gitlab/ -type f -name $FILENAME)"
# Todo check $EE is not empty
echo "Gitlab backup repo file to be uploaded is: " $EE >> $LOGFILE
/usr/local/bin/aws s3 cp ${EE} s3://meow/ >>  $LOGFILE

#---------1. GITLAB uploads backup ---------

echo "Start - gitlab uploads folder to s3"  >> $LOGFILE

BAK_DES=/mnt/data/gitlab/backups/
BAK_SOURCES=/mnt/data/gitlab/uploads/

BAK_DATE=`date +%F`
BAK_DATETIME=`date +%F-%H%M`
BAK_FOLDER=${BAK_DEST}/${BAK_DATE}
BAK_FILE=${BAK_DES}uploads-${BAK_DATETIME}.tar.gz

echo "creating tar.gz file at " ${BAK_FILE} >> $LOGFILE

tar czPf ${BAK_FILE} ${BAK_SOURCES}

echo "Now uploading gitlab uploads folder to s3"

/usr/local/bin/aws s3 cp ${BAK_FILE} s3://meow/ >>  $LOGFILE

#---------3. Delete old files ---------

echo 'Deleting backup older than '${KEEP_DAYS}' days'  >> $LOGFILE
find /mnt/data/gitlab/backups/ -type f -name '*.tar' -mtime +3 -exec rm {} \;
find /mnt/data/gitlab/backups/ -type f -name '*.tar.gz' -mtime +3 -exec rm {} \;

#---------4. Send email to me to make me smile ---------

SENDGRID_API_KEY="long-key-here"
BAK_DATETIME=`date +%F-%H:%M`
SUBJECT="Gitlab backup to succcessful: ${BAK_DATETIME}"
REQUEST_DATA='{"personalizations": [{ 
     "to": [{ "email": "foo@foo.bar" }],
     "subject": "'"$SUBJECT"'"
  }],
  "from": {
      "email": "foo@foo.bar",
      "name": "Code.spritle.com" 
  },
  "content": [{
      "type": "text/plain",
      "value": "Keep smiling"
  }]
}';

curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
 -H "Authorization: Bearer $SENDGRID_API_KEY" \
 -H "Content-Type: application/json" \
 -d "$REQUEST_DATA"

echo "Sent email notification via sendgrid" >> $LOGFILE
echo "***************END***************************" >> $LOGFILE

This script seems good. But its missing the custom configurations like where to make changes if I want to use my own s3 bucket with accesskey and secret

The aws cli should automatically pick that up if it's configured correctly. You never place those or any other credentials in a executing script or environment variables where they might be exposed to others accessing the machine.

Why you don't use native S3 integration in Gitlab for uploading backup into cloud? 🤔

@Rohlik The very idea I created this custom script was my gitlab instance auto backup to s3 was not working and was failing for unknown reasons.