POC Vault Restore

01-vault-restore-poc.txt

To restore a filesystem-backed Vault instance:

  1. Shut down running Vault process (pkill vault)
  2. Make backup to new location (cp -r /original-storage /new-storage)
  3. Write a new config file to point to /new-storage
  4. Start new Vault process (vault server -config=new-config-file.hcl)
  5. DO NOT run `vault init`
  6. ONLY RUN `vault unseal <key1>`, etc...

02-vault-restore-poc.sh

#!/bin/bash

die() { echo "ERROR: $@" >&2; pkill vault; exit 1; }

vault version

mkdir /tmp/vault-test || die "Could not make /tmp/vault-test directory"
cd /tmp/vault-test || die "Could not change to /tmp/vault-test directory"

rm -rf orig/ orig.* restore/ restore.*

echo
echo Creating orig.conf:
tee orig.conf <<EOF
backend "file" {
  path = "$(pwd)/orig"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}
EOF

echo
echo -n "Starting vault... "
vault server -config=$(pwd)/orig.conf &>orig.log &
# pause for startup
sleep 2
echo OK

export VAULT_ADDR=http://127.0.0.1:8200

echo "Initializing vault:"
{
    vault init 2>&1 \
       || die "Could not init orig vault"
} |tee orig.init.out

echo
echo -n "Finding token and keys... "
read key1 key2 key3 token < <( echo $( grep -E '^(Key [123]|Initial Root Token):' orig.init.out |cut -d: -f2- ) )
echo OK

echo
echo "Checking vault status:"
vault status \
    && { echo; die "SURPRISE: Vault is unsealed"; } \
    || { echo; echo "OK: Vault is still sealed"; }

echo
echo "Unsealing the vault:"
set -x
vault unseal $key1
vault unseal $key2
vault unseal $key3
set +x

echo
echo "Checking vault status:"
vault status \
    && { echo; echo "OK: Vault is unsealed"; } \
    || { echo; die "Vault is still sealed"; }

echo
export VAULT_TOKEN=$token

echo "Writing secrets:"
mysecret_in=abc123
combination_in=12345

set -x
vault write secret/test/one mysecret=$mysecret_in
vault write secret/example combination=$combination_in
set +x

echo
echo "Reading secrets:"
set -x
mysecret_out=$( vault read -field mysecret secret/test/one )
combination_out=$( vault read -field combination secret/example )
set +x
echo

if [[ $mysecret_in == $mysecret_out ]] && [[ $combination_in == $combination_out ]]
then
    echo "OK: The secrets are correct so far"
else
    die "The secrets are incorrect"
fi


echo
echo -n "Shutting down vault... "
pkill vault
sleep 2
echo OK

echo
echo -n "Making backup of orig/ to restore/ ... "
cp -r orig restore
echo OK

echo
echo Creating restore.conf:
tee restore.conf <<EOF
backend "file" {
  path = "$(pwd)/restore"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}
EOF

echo
echo -n "Starting restored vault... "
vault server -config=$(pwd)/restore.conf &>restore.log &
# pause for startup
sleep 2
echo OK

echo
echo "Attempting vault init:"
{
    vault init 2>&1 \
        && die "Was able to init the restore vault, this should not happen" \
        || echo "AS EXPECTED: Could not init restore vault" >&2
} |tee restore.init.out

echo
echo "Checking vault status:"
vault status \
    && { echo; die "SURPRISE: Vault is unsealed"; } \
    || { echo; echo "OK: Vault is still sealed"; }

echo
echo "Unsealing the restore vault using the original keys:"
set -x
vault unseal $key1
vault unseal $key2
vault unseal $key3
set +x

echo
echo "Checking vault status:"
vault status \
    && { echo; echo "OK: Vault is unsealed"; } \
    || { echo; die "Vault is still sealed"; }


echo
echo "Reading secrets:"
set -x
mysecret_restore=$( vault read -field mysecret secret/test/one )
combination_restore=$( vault read -field combination secret/example )
set +x
echo

if [[ $mysecret_in == $mysecret_restore ]] && [[ $combination_in == $combination_restore ]]
then
    echo "YAY: The secrets are correct in the restored vault!"
else
    die "The secrets are incorrect"
fi

echo
echo -n "Shutting down vault... "
pkill vault
sleep 2
echo OK

03-transcript.txt

$ bash 02-vault-restore-poc.sh 
Vault v0.5.2

Creating orig.conf:
backend "file" {
  path = "/tmp/vault-test/orig"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

Starting vault... OK
Initializing vault:
Key 1: 4c22404bf7ddc011e8a4c21c5bcceff1afc1856e4d8b5689d69c616e7c9aaff001
Key 2: 840cef75119c0338f4723aa667427f405d78aa430b1d1a67184470ec54ba968a02
Key 3: d697d3bc7c9821e4248645c3f78d67c5b931a36845a1e28c669359e2ccc2c62e03
Key 4: c99ec094c87c997f7c7d5788e2bc2a885ee01012755cba65f88b8b15c5c424f104
Key 5: 9b05fc5da578bba3ac8928ed7273320dbaa919393be0428e865ca21b5dbc745505
Initial Root Token: 56b24a14-c3bd-e814-a5cd-d0bac1fcecc5

Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.

Finding token and keys... OK

Checking vault status:
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: false

OK: Vault is still sealed

Unsealing the vault:
+ vault unseal 4c22404bf7ddc011e8a4c21c5bcceff1afc1856e4d8b5689d69c616e7c9aaff001
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
+ vault unseal 840cef75119c0338f4723aa667427f405d78aa430b1d1a67184470ec54ba968a02
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
+ vault unseal d697d3bc7c9821e4248645c3f78d67c5b931a36845a1e28c669359e2ccc2c62e03
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
+ set +x

Checking vault status:
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: false

OK: Vault is unsealed

Writing secrets:
+ vault write secret/test/one mysecret=abc123
Success! Data written to: secret/test/one
+ vault write secret/example combination=12345
Success! Data written to: secret/example
+ set +x

Reading secrets:
++ vault read -field mysecret secret/test/one
+ mysecret_out=abc123
++ vault read -field combination secret/example
+ combination_out=12345
+ set +x

OK: The secrets are correct so far

Shutting down vault... OK

Making backup of orig/ to restore/ ... OK

Creating restore.conf:
backend "file" {
  path = "/tmp/vault-test/restore"
}

# no need for setting this up in testing
disable_mlock = true

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

Starting restored vault... OK

Attempting vault init:
Error initializing Vault: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:

* Vault is already initialized
AS EXPECTED: Could not init restore vault

Checking vault status:
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: false

OK: Vault is still sealed

Unsealing the restore vault using the original keys:
+ vault unseal 4c22404bf7ddc011e8a4c21c5bcceff1afc1856e4d8b5689d69c616e7c9aaff001
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
+ vault unseal 840cef75119c0338f4723aa667427f405d78aa430b1d1a67184470ec54ba968a02
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
+ vault unseal d697d3bc7c9821e4248645c3f78d67c5b931a36845a1e28c669359e2ccc2c62e03
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
+ set +x

Checking vault status:
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: false

OK: Vault is unsealed

Reading secrets:
++ vault read -field mysecret secret/test/one
+ mysecret_restore=abc123
++ vault read -field combination secret/example
+ combination_restore=12345
+ set +x

YAY: The secrets are correct in the restored vault!

Shutting down vault... OK

Need to seal vault before stopping(L98 ) else script will exit at L141