gistfile1.txt

==
[23:50:35]:Failed to create backup.
[23:50:35]:<em style='color: #f3990b;'>10.0.8.18</em>:<em style='color: #0b33b5;'>/root/backups/BACKUP-0</em>: Preferring the global retention setting: 31 days (be deleted on 2020-12-13).
[23:50:34]:<em style='color: #d35400;'>10.0.8.15:27017</em>: Evaluating command by mongo client on the remote host: sh.setBalancerState(true);
[23:50:34]:<em style='color: #d35400;'>10.0.8.15:27017</em>: Enable the balancer process by issuing sh.setBalancerState(true).
[23:50:34]:Backup finished with failure.
[23:50:34]:Mongo node 10.0.8.18:27018 status: 1 role: shardsvr: Backup command failed.
Starting backup '2020-11-12T14:50:34Z'
StdErr:

Error starting backup: another operation in progress, restore/2020-11-12T14:46:52.922150265Z [rs2/10.0.8.21:27018]
s
[23:50:34]:<em style='color: #f3990b;'>10.0.8.18</em>:       Error 
Error starting backup: another operation in progress, restore/2020-11-12T14:46:52.922150265Z [rs2/10.0.8.21:27018]
.
[23:50:34]:<em style='color: #f3990b;'>10.0.8.18</em>:      Output Starting backup '2020-11-12T14:50:34Z'.
[23:50:34]:<em style='color: #f3990b;'>10.0.8.18</em>: Command was <strong style='color: orange;'>bash /etc/pbm-agent-27018/pbm.sh backup</strong>.
[23:50:34]:<em style='color: #f3990b;'>10.0.8.18</em>: Execution failed with return code 1.
[23:50:32]:<em style='color: #f3990b;'>10.0.8.18</em>: Running pbm backup.
[23:50:31]:<em style='color: #f3990b;'>10.0.8.18</em>: Creating pbm executer script '/etc/pbm-agent-27018/pbm.sh'.
[23:50:31]:Going to run pbm backup on host '<em style='color: #f3990b;'>10.0.8.18</em>'.
[23:50:31]:All PBMAgent node is running.
[23:50:31]:PBMAgent looks being available on all data nodes.
[23:50:31]:Collecting all available data nodes to create a list of missing PBMAgent nodes.
[23:50:29]:<em style='color: #d35400;'>10.0.8.15:27017</em>: Evaluating command by mongo client on the remote host: sh.stopBalancer();
[23:50:29]:<em style='color: #d35400;'>10.0.8.15:27017</em>: Disable the balancer process by issuing sh.stopBalancer().
[23:50:29]:Created backup record, id is 8.
[23:50:28]:Elected host 10.0.8.18:27018 for creating the backup.
[23:50:28]:The username is '<strong style='color: #59a449;'>root</strong>'.
[23:50:28]:Cluster ID is 2.
[23:50:28]:Using SSH credentials from cluster.
Job spec: {
    "command": "backup",
    "group_id": 1,
    "group_name": "admins",
    "user_id": 1,
    "user_name": "admin@localhost.xyz",
    "job_data": 
    {
        "backup_method": "percona-backup-mongodb",
        "clusterId": "2"
    }
}

==
ccurrent topology: Type: ReplicaSetNoPrimary
Servers:
Addr: 10.0.8.17:27019, Type: RSSecondary, State: Connected, Average RTT: 787594
Addr: 10.0.8.15:27019, Type: RSSecondary, State: Connected, Average RTT: 1338330
Addr: 10.0.8.16:27019, Type: RSSecondary, State: Connected, Average RTT: 945863

2020/11/12 14:44:16 [ERROR] wrting log: db: server selection error: server selection timeout
current topology: Type: ReplicaSetNoPrimary
Servers:
Addr: 10.0.8.15:27019, Type: RSSecondary, State: Connected, Average RTT: 1073953
Addr: 10.0.8.16:27019, Type: RSSecondary, State: Connected, Average RTT: 1143222
Addr: 10.0.8.17:27019, Type: RSSecondary, State: Connected, Average RTT: 1477306
, <nil>, entry: 2020-11-12T14:43:46.000+0000 [ERROR] listening commands: watch the cmd stream: server selection error: server selection timeout
current topology: Type: ReplicaSetNoPrimary
Servers:
Addr: 10.0.8.17:27019, Type: RSSecondary, State: Connected, Average RTT: 787594
Addr: 10.0.8.15:27019, Type: RSSecondary, State: Connected, Average RTT: 1338330
Addr: 10.0.8.16:27019, Type: RSSecondary, State: Connected, Average RTT: 945863
2020-11-12T14:44:16.000+0000 [ERROR] listening commands: watch the cmd stream: server selection error: server selection timeout
current topology: Type: ReplicaSetNoPrimary
Servers:
Addr: 10.0.8.16:27019, Type: RSSecondary, State: Connected, Average RTT: 1143222
Addr: 10.0.8.17:27019, Type: RSSecondary, State: Connected, Average RTT: 1477306
Addr: 10.0.8.15:27019, Type: RSSecondary, State: Connected, Average RTT: 1073953

2020-11-12T14:44:46.000+0000 [INFO] pitr: streaming started from 2020-11-12 14:32:06 +0000 UTC / 1605191526
2020-11-12T14:46:07.000+0000 [INFO] pitr: got done signal, stopping
2020-11-12T14:46:45.000+0000 [INFO] pitr: created chunk 2020-11-12T14:32:06 - 2020-11-12T14:46:36
2020-11-12T14:46:45.000+0000 [INFO] pitr: pausing/stopping with last_ts 2020-11-12 14:46:36 +0000 UTC
2020-11-12T14:46:53.000+0000 [INFO] got command restore [name: 2020-11-12T14:46:52.922150265Z, backup name: 2020-11-12T07:21:00Z] <ts: 1605192412>
2020-11-12T14:46:53.000+0000 [INFO] restore/2020-11-12T07:21:00Z: restore started
2020-11-12T14:46:56.447+0000    setting number of parallel collections to number of parallel collections in archive (3)
2020-11-12T14:46:56.449+0000    preparing collections to restore from
2020-11-12T14:46:56.720+0000    reading metadata for admin.pbmRUsers from archive on stdin
2020-11-12T14:46:56.748+0000    restoring admin.pbmRUsers from archive on stdin
2020-11-12T14:46:57.201+0000    restoring indexes for collection admin.pbmRUsers from metadata
2020-11-12T14:46:57.233+0000    finished restoring admin.pbmRUsers (2 documents, 0 failures)
2020-11-12T14:46:57.234+0000    reading metadata for admin.pbmRRoles from archive on stdin
2020-11-12T14:46:57.257+0000    restoring admin.pbmRRoles from archive on stdin
2020-11-12T14:46:57.709+0000    restoring indexes for collection admin.pbmRRoles from metadata
2020-11-12T14:46:57.789+0000    finished restoring admin.pbmRRoles (1 document, 0 failures)
2020-11-12T14:46:57.000+0000 [INFO] restore/2020-11-12T07:21:00Z: mongorestore finished
2020-11-12T14:51:29.000+0000 [INFO] restore/2020-11-12T07:21:00Z: starting oplog replay
2020-11-12T14:51:29.000+0000 [INFO] restore/2020-11-12T07:21:00Z: oplog replay finished on {0 0}
2020-11-12T14:51:29.000+0000 [INFO] restore/2020-11-12T07:21:00Z: restoring users and roles
2020-11-12T14:51:30.000+0000 [INFO] restore/2020-11-12T07:21:00Z: restore finished successfully
2020-11-12T14:51:35.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:52:04.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:52:34.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:53:04.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:53:34.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:54:03.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:54:33.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:55:03.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:55:32.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:56:02.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:56:32.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:57:02.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR
2020-11-12T14:57:31.000+0000 [ERROR] pitr: defining starting point for the backup: no backup found after the restored 2020-11-12T07:21:00Z, a new backup is required to resume PITR