How to copy production database on AWS RDS(postgresql) to local development database.

how-to-copy-aws-rds-to-local.md

1. Change your database RDS instance security group to allow your machine to access it.
   • Add your ip to the security group to acces the instance via Postgres.
2. Make a copy of the database using pg_dump
   • $ pg_dump -h <public dns> -U <my username> -f <name of dump file .sql> <name of my database>
   • you will be asked for postgressql password.
   • a dump file(.sql) will be created
3. Restore that dump file to your local database.
   • but you might need to drop the database and create it first
   • $ psql -U <postgresql username> -d <database name> -f <dump file that you want to restore>
   • the database is restored
4. pg_restore -h <host> -U <username> -c -d <database name> <filename to be restored>

#####ref

1. http://stackoverflow.com/questions/31881786/how-to-pg-dump-an-rds-postgres-database
2. http://www.thegeekstuff.com/2009/01/how-to-backup-and-restore-postgres-database-using-pg_dump-and-psql/

Just a note: if you're using a custom VPC, make sure you enable "Publicly Accessible" in your DB instance and that your VPC has an internet gateway (by attaching the gateway to the VPC ID).

Future me: also make sure your route table is actually pointing to your internet gateway: https://aws.amazon.com/premiumsupport/knowledge-center/instance-vpc-troubleshoot/ because last time it was pointing to an old one that didn't allow me to connect to anything.

I get this error "pg_dump: error: could not open output file "test_dump_20210121.sql": Permission denied"

Any ideas?

Hey @burakuytun-rightsoft it might be because your current user doesn't have the right permissions. Try running sudo chmod 666 test_dump_20210121.sql.

Thanks for this. #4 was not needed.
The extra step I had to do before restoring the dump, was create the DB, as well as the users and roles associated with it.
Adding a user for rdsadmin is necessary if it's the owner of any objects in your DB. If not, then you should be fine ignoring the error.

PS. Depending on the size (and purpose) of your DB, it might make sense to change the location of postgres' data directory

Thanks @kinduff :)

👍

If you want to restore a dump into a differently named database, make sure not use the --create option on pg_dump as it will force create the DB under that name.

If you use the --format c flag you can use pg_restore and save some bandwidth.

psql -h localhost -U user -d postgres -c "DROP DATABASE IF EXISTS newdb (FORCE)"
psql -h localhost -U user -d postgres -c "CREATE DATABASE newdb WITH OWNER = postgres ENCODING = 'UTF8'"
pg_dump --format c --blobs --no-owner --no-acl --quote-all-identifiers --verbose --dbname=postgresql://user:password@domain.rds.amazonaws.com/databasename --file outputfile.bak
pg_restore --verbose -h localhost -U user -d newdb outputfile.bak

Could add here the step to close all connections to the current running DB, before deletion:
psql -h localhost -U user -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='newdb'"

I am getting "AccessShareLock" on a few tables when I try to generate the dump as there are multiple active connections to the db. Is there any way to clone it to a read replica even when the connections are active?

@swapnilj01 You could try passing the --serializable-deferrable option to pg_dump. That gives you a consistent read-only snapshot of the database. I'm not entirely sure how that interacts with locks but I expect it should work.