From the primary server, issue a takeover hadr on each db as the db2inst1 user. Below is a for loop that I used to do make sure everything is where it should be:
for db in $(db2 list db directory|grep alias |awk -F\= '{print $NF}')
do
db2 takeover hadr on db ${db}
done
You can use the 'lssam' command to validate:
Online IBM.ResourceGroup:db2_db2inst1_db2inst1_OS3DB-rg Nominal=Online
'- Online IBM.Application:db2_db2inst1_db2inst1_OS3DB-rs
|- Online IBM.Application:db2_db2inst1_db2inst1_OS3DB-rs:SD00030LINVDAT
'- Offline IBM.Application:db2_db2inst1_db2inst1_OS3DB-rs:SD00031LINVDAT
The server "Online" is the active node.
At this point you need to stop HADR on the STANDBY server:
db2 deactivate db <db>
db2 stop hadr on db <db>
Next you need to stop HADR on the PRIMARY server:
db2 stop hadr on db <db>
db2 deactivate db <db>
~ Yes, those commands are inversed from what you did on the standb... ~
Now you can begin running the restores on the PRIMARY server:
db2 restore db <db> from /path/to/backup taken at <timestamp>
For ECM the db path is usually /db2backups/backup/YYYMMDD, the TS can be pulled from the backup file:
FNGCDDB.0.db2inst1.DBPART000.20171023030003.001
This is the 20171023030003 portion of the file name.
When the restore completes, you need roll the logs forward:
db2 rollforward db <db> to end of logs
db2 rollforward db os1db complete
### Another alternative is to use
db2 rollforward db <db> to end of backup
db2 rollforward db <db> complete
Sometimes you will run into an issue where logs are missing:
[db2inst1@SD00030LINVDAT ~]$ db2 rollforward db os1db to end of logs
SQL1273N An operation reading the logs on database "OS1DB" cannot continue
because of a missing log file "S0003403.LOG" on database partition "0" and log
stream "0".
This generally means logs are missing. We need to understand how we can
This can take some time depending on the logs, and size of logs. When this completes, you will need to run an offline backup.
mkdir -p /db2backups/backup/YYYYMMDD
db2 backup db <db> to /db2backups/backup/YYYYMMDD
You will need to copy the offline backup to the standby server; I found that mounting a common export from the NFS server and copying it that way to be the easiest. You could just scp it as well.
On the Standby server you need to restore that offline db. You also have to update the db cfg make sure the hostnames are correct. Then you start hadr on the standby.
db2 restore db <db> from /db2backups/backup/YYYYMMDD taken at <TS>
db2 update db cfg for <db> using HADR_LOCAL_HOST SD00031LINVDAT.island.local
db2 update db cfg for <db> using HADR_REMOTE_HOST SD00030LINVDAT.island.local
db2 start hadr on db <db> as standby
On the Primary, start HADR on the db:
db2 start hadr on db os2db as primary
You might have to unlock the resource group, but give this 5-10 minutes first.
### These are the steps I took to restore 'FNGCDDB'
# standby
db2 deactivate db fngcddb
db2 stop hadr on db fngcddb
# primary
db2 stop hadr on db fngcddb
db2 deactivate db fngcddb
db2 restore db fngcddb from /db2backups/backup/20171023 taken at 20171023030003
cp -R /mnt/db2_arclog/fngcddb /db2_arclog
db2 rollforward db fngcddc to end of backup
db2 rollforward db fngcddc complete
db2 backup db fngcddb to /db2backups/backup/20171109
cp -R /db2backups/backup/20171109 /mnt/backup
#standby
cp -R /mnt/backup/20171109 /db2backups/backup
db2 restore db fngcddb from /db2backups/backup/20171109 taken at 20171109102020
db2 update db cfg for fngcddb using HADR_LOCAL_HOST SD00031LINVDAT.island.local
db2 update db cfg for fngcddb using HADR_REMOTE_HOST SD00030LINVDAT.island.local
db2 start hadr on db fngcddb as standby
# primary
db2 start hadr on db fngcddb as primary