POC Vault Restore

01-vault-restore-poc.txt

To restore a filesystem-backed Vault instance:

  1. Seal existing vault (vault seal)
  2. Shut down running Vault process (pkill vault)
  3. Make backup to new location (cp -r /original-storage /new-storage)
  4. Write a new config file to point to /new-storage
  5. Start new Vault process (vault server -config=new-config-file.hcl)
  6. DO NOT run `vault init`
  7. ONLY RUN `vault unseal <key1>`, etc...

02-vault-restore-poc.sh

#!/bin/bash

die() { echo "ERROR: $@" >&2; pkill vault; exit 1; }

vault version

mkdir /tmp/vault-test || die "Could not make /tmp/vault-test directory"
cd /tmp/vault-test || die "Could not change to /tmp/vault-test directory"

rm -rf orig/ orig.* restore/ restore.*

echo
echo Creating orig.conf:
tee orig.conf <<EOF
backend "file" {
  path = "$(pwd)/orig"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}
EOF

echo
echo -n "Starting vault... "
vault server -config=$(pwd)/orig.conf &>orig.log &
# pause for startup
sleep 2
echo OK

export VAULT_ADDR=http://127.0.0.1:8200

echo "Initializing vault:"
{
    vault init 2>&1 \
       || die "Could not init orig vault"
} |tee orig.init.out

echo
echo -n "Finding token and keys... "
read key1 key2 key3 token < <( echo $( grep -E '^(Unseal Key [123]|Initial Root Token):' orig.init.out |cut -d: -f2- ) )
echo OK

echo
echo "Checking vault status:"
vault status \
    && { echo; die "SURPRISE: Vault is unsealed"; } \
    || { echo; echo "OK: Vault is still sealed"; }

echo
vault status
echo "Unsealing the vault:"
set -x
vault unseal $key1
vault unseal $key2
vault unseal $key3
set +x
vault status

echo
echo "Checking vault status:"
vault status \
    && { echo; echo "OK: Vault is unsealed"; } \
    || { echo; die "Vault is still sealed"; }

echo
export VAULT_TOKEN=$token

echo "Writing secrets:"
mysecret_in=abc123
combination_in=12345

set -x
vault write secret/test/one mysecret=$mysecret_in
vault write secret/example combination=$combination_in
set +x

echo
echo "Reading secrets:"
set -x
mysecret_out=$( vault read -field mysecret secret/test/one )
combination_out=$( vault read -field combination secret/example )
set +x
echo

if [[ $mysecret_in == $mysecret_out ]] && [[ $combination_in == $combination_out ]]
then
    echo "OK: The secrets are correct so far"
else
    die "The secrets are incorrect"
fi


echo -n "Sealed the vault"
vault seal
vault status
sleep 2

echo
echo -n "Shutting down vault... "
pkill vault
sleep 2
echo OK

echo
echo -n "Making backup of orig/ to restore/ ... "
cp -r orig restore
echo OK

echo
echo Creating restore.conf:
tee restore.conf <<EOF
backend "file" {
  path = "$(pwd)/restore"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}
EOF

echo
echo -n "Starting restored vault... "
vault server -config=$(pwd)/restore.conf &>restore.log &
# pause for startup
sleep 2
echo OK

echo
echo "Attempting vault init:"
{
    vault init 2>&1 \
        && die "Was able to init the restore vault, this should not happen" \
        || echo "AS EXPECTED: Could not init restore vault" >&2
} |tee restore.init.out

echo
echo "Checking vault status:"
vault status \
    && { echo; die "SURPRISE: Vault is unsealed"; } \
    || { echo; echo "OK: Vault is still sealed"; }

echo
echo "Unsealing the restore vault using the original keys:"
set -x
vault unseal $key1
vault unseal $key2
vault unseal $key3
set +x

echo
echo "Checking vault status:"
vault status \
    && { echo; echo "OK: Vault is unsealed"; } \
    || { echo; die "Vault is still sealed"; }


echo
echo "Reading secrets:"
set -x
mysecret_restore=$( vault read -field mysecret secret/test/one )
combination_restore=$( vault read -field combination secret/example )
set +x
echo

if [[ $mysecret_in == $mysecret_restore ]] && [[ $combination_in == $combination_restore ]]
then
    echo "YAY: The secrets are correct in the restored vault!"
else
    die "The secrets are incorrect"
fi

echo -n "Sealed the vault"
vault seal
vault status
sleep 2

echo
echo -n "Shutting down vault... "
pkill vault
sleep 2
echo OK

03-transcript.txt

Vault v0.9.0 ('bdac1854478538052ba5b7ec9a9ec688d35a3335')

Creating orig.conf:
backend "file" {
  path = "/tmp/vault-test/orig"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

Starting vault... OK
Initializing vault:
Unseal Key 1: bzghEw3rac3Zau7pdVe7KoGcgOboULZzMlC4lrWdwwAZ
Unseal Key 2: Vr5oChLsXT2mJuv+CH5QzNFNppP7mEY6Bn23PG59pBHh
Unseal Key 3: LLiIU/aVrOW5aYp3WCcGxeYCv9QLzDUzvPl8W8NjBaZL
Unseal Key 4: BScdwwhy/CzNRZS7qe5fry4zPaXcuzm1oqnHdvv8vzJg
Unseal Key 5: rm72OuhRJHA9dwIgIf/GIw4d6LHnhhfGPGTCvVHdbhud
Initial Root Token: 82c489ad-6798-8d2f-edbd-01002052dfd4

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your vault will remain permanently sealed.

Finding token and keys... OK

Checking vault status:
Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0

High-Availability Enabled: true
    Mode: sealed

OK: Vault is still sealed

Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0

High-Availability Enabled: true
    Mode: sealed
Unsealing the vault:
+ ./vault unseal bzghEw3rac3Zau7pdVe7KoGcgOboULZzMlC4lrWdwwAZ
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: 113ef7f2-cbf5-3bed-77d9-2cd76c532728
+ ./vault unseal Vr5oChLsXT2mJuv+CH5QzNFNppP7mEY6Bn23PG59pBHh
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
Unseal Nonce: 113ef7f2-cbf5-3bed-77d9-2cd76c532728
+ ./vault unseal LLiIU/aVrOW5aYp3WCcGxeYCv9QLzDUzvPl8W8NjBaZL
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
+ set +x
Type: shamir
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0
Cluster Name: vault-cluster-2e6456f6
Cluster ID: 085e65a7-3547-a4cb-26d2-3f0052484299

High-Availability Enabled: false

Checking vault status:
Type: shamir
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0
Cluster Name: vault-cluster-2e6456f6
Cluster ID: 085e65a7-3547-a4cb-26d2-3f0052484299

High-Availability Enabled: false

OK: Vault is unsealed

Writing secrets:
+ ./vault write secret/test/one mysecret=abc123
Success! Data written to: secret/test/one
+ ./vault write secret/example combination=12345
Success! Data written to: secret/example
+ set +x

Reading secrets:
++ ./vault read -field mysecret secret/test/one
+ mysecret_out=abc123
++ ./vault read -field combination secret/example
+ combination_out=12345
+ set +x

OK: The secrets are correct so far
Sealed the vaultVault is now sealed.
Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0

High-Availability Enabled: true
    Mode: sealed



Shutting down vault... OK

Making backup of orig/ to restore/ ... OK

Creating restore.conf:
backend "file" {
  path = "/tmp/vault-test/restore"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

Starting restored vault... OK

Attempting vault init:
Error initializing Vault: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:

* Vault is already initialized
AS EXPECTED: Could not init restore vault

Checking vault status:
Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0

High-Availability Enabled: true
    Mode: sealed

OK: Vault is still sealed

Unsealing the restore vault using the original keys:
+ ./vault unseal bzghEw3rac3Zau7pdVe7KoGcgOboULZzMlC4lrWdwwAZ
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: 9780ee37-3321-d94d-6dc3-992bcdd17a02
+ ./vault unseal Vr5oChLsXT2mJuv+CH5QzNFNppP7mEY6Bn23PG59pBHh
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
Unseal Nonce: 9780ee37-3321-d94d-6dc3-992bcdd17a02
+ ./vault unseal LLiIU/aVrOW5aYp3WCcGxeYCv9QLzDUzvPl8W8NjBaZL
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
+ set +x

Checking vault status:
Type: shamir
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0
Cluster Name: vault-cluster-2e6456f6
Cluster ID: 085e65a7-3547-a4cb-26d2-3f0052484299

High-Availability Enabled: false

OK: Vault is unsealed

Reading secrets:
++ ./vault read -field mysecret secret/test/one
+ mysecret_restore=abc123
++ ./vault read -field combination secret/example
+ combination_restore=12345
+ set +x

YAY: The secrets are correct in the restored vault!
Sealed the vaultVault is now sealed.
Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.0

High-Availability Enabled: true
    Mode: sealed



Shutting down vault... OK