Skip to content

Instantly share code, notes, and snippets.

@billimek
Last active November 3, 2023 14:25
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save billimek/2f2d1ef23e38442daef4b3bafc400fef to your computer and use it in GitHub Desktop.
Save billimek/2f2d1ef23e38442daef4b3bafc400fef to your computer and use it in GitHub Desktop.
vault transit auto-unseal

Dual vault servers that unseal each-other with transit seal type

Instructions inspired from auto unseal with transit guide

Prerequisites

  • Docker
  • vault CLI

1. Bootstrap first vault server

# make persistent directories and set permissions
mkdir -p vault/config
mkdir -p vault/data
sudo chown 100:100 vault/data

# make initial config file
tee vault/config/config.hcl <<EOF
storage "file" {
    path = "/vault/data"
}

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}
EOF

# run vault server
docker run -d --name vault --cap-add=IPC_LOCK -p 8200:8200 -v $(pwd)/vault/config:/vault/config -v $(pwd)/vault/data:/vault/data vault server

# initialize vault & make note of root token and unseal key
VAULT_ADDR=http://127.0.0.1:8200 vault operator init -n 1 -t 1

# unseal the vault server
VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal <unseal key from above>

2. Configure first vault server to act as a transit server

# login to the vault server with the root token
VAULT_ADDR=http://127.0.0.1:8200 vault login <root token from above>

# Enable the transit secrets engine
VAULT_ADDR=http://127.0.0.1:8200 vault secrets enable transit

# Create a key named 'autounseal'
VAULT_ADDR=http://127.0.0.1:8200 vault write -f transit/keys/autounseal

# Create a policy file
tee autounseal.hcl <<EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}
EOF

# Create an 'autounseal' policy
VAULT_ADDR=http://127.0.0.1:8200 vault policy write autounseal autounseal.hcl

# Create a client token with autounseal policy attached and response wrap it with TTL of 600 seconds.
VAULT_ADDR=http://127.0.0.1:8200 vault token create -policy="autounseal" -wrap-ttl=600

# make note of the "wrapping_token" from the output above

# unwrap the autounseal token and capture the client token
VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=<wrapping_token> vault unwrap

# make note of the "token" - this will be the client token used for vault instance #2

3. Bootstrap second vault server

# make persistent directories and set permissions
mkdir -p vault2/config
mkdir -p vault2/data
sudo chown 100:100 vault2/data

# make initial config file
tee vault2/config/config.hcl <<EOF
storage "file" {
    path = "/vault/data"
}

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}

seal "transit" {
  address = "http://<ip adress of the host where the first vault server is running>:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}
EOF

# run vault server
docker run -d --name vault2 --cap-add=IPC_LOCK -p 8100:8200 -e VAULT_TOKEN="<client token obtained from vault server 1 above>" -v $(pwd)/vault2/config:/vault/config -v $(pwd)/vault2/data:/vault/data vault server

# initialize vault & make note of root token and unseal key
VAULT_ADDR=http://127.0.0.1:8100 vault operator init -recovery-shares=1 -recovery-threshold=1

# unseal the vault server
VAULT_ADDR=http://127.0.0.1:8100 vault operator unseal <unseal key from above>

4. Configure second vault server to act as a transit server

# login to the second vault server with the root token
VAULT_ADDR=http://127.0.0.1:8100 vault login <root token from above>

# Enable the transit secrets engine
VAULT_ADDR=http://127.0.0.1:8100 vault secrets enable transit

# Create a key named 'autounseal'
VAULT_ADDR=http://127.0.0.1:8100 vault write -f transit/keys/autounseal

# Create a policy file
tee autounseal.hcl <<EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}
EOF

# Create an 'autounseal' policy
VAULT_ADDR=http://127.0.0.1:8100 vault policy write autounseal autounseal.hcl

# Create a client token with autounseal policy attached and response wrap it with TTL of 600 seconds.
VAULT_ADDR=http://127.0.0.1:8100 vault token create -policy="autounseal" -wrap-ttl=600

# make note of the "wrapping_token" from the output above

# unwrap the autounseal token and capture the client token
VAULT_ADDR=http://127.0.0.1:8100 VAULT_TOKEN=<wrapping_token> vault unwrap

# make note of the "token" - this will be the client token used for vault instance #2

5. Add transit seal type to first vault server

# append transit seal type to first vault server config file
tee -a vault/config/config.hcl <<EOF

seal "transit" {
  address = "http://<ip adress of the host where the second vault server is running>:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}
EOF

# shut-down first vault server
docker rm -f vault

# restart first vault server with the client token obtained from the previous step
docker run -d --name vault --cap-add=IPC_LOCK -p 8200:8200 -e VAULT_TOKEN="<client token obtained from vault server 2 above>" -v $(pwd)/vault/config:/vault/config -v $(pwd)/vault/data:/vault/data vault server

# unseal the first vault server again
VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal <unseal key for first vault server>

# login to the first vault server with the root token
VAULT_ADDR=http://127.0.0.1:8200 vault login <root token for first vault server>

# migrate the unseal type from shamir to the transit type this was just configured for
VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal -migrate

6. Profit!

Now the two vault servers will unseal each-other.

@patsevanton
Copy link

patsevanton commented May 10, 2022

Thanks for gist. I have error

seal.transit: unable to renew token, disabling renewal: err="Put \"http://192.168.10.8:8200/v1/auth/token/renew-self\": EOF"

How fix it?

ubuntu@vault0:~$ sudo su -
root@vault0:~# mkdir -p vault/config
root@vault0:~# mkdir -p vault/data
root@vault0:~# sudo chown 100:1000 vault/data


root@vault0:~# tee vault/config/config.hcl <<EOF
> storage "file" {
>     path = "/vault/data"
> }
>
> listener "tcp" {
>   tls_disable = 1
>   address = "[::]:8200"
>   cluster_address = "[::]:8201"
> }
> EOF
storage "file" {
    path = "/vault/data"
}

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}


root@vault0:~# docker run -d --name vault --cap-add=IPC_LOCK -p 8200:8200 -v $(pwd)/vault/config:/vault/config -v $(pwd)/vault/data:/vault/data vault server
Unable to find image 'vault:latest' locally
latest: Pulling from library/vault
8663204ce13b: Pull complete
4ef4980bef8c: Pull complete
ecc0822110ac: Pull complete
94096efdbbad: Pull complete
c572f71443d7: Pull complete
Digest: sha256:34fa80e67ccc4b7c78d7ed08bff7e2049ab9e5140d04d2223f72168fc05672dc
Status: Downloaded newer image for vault:latest
affe1f05be3b5d68c52b20a77d8899d837930e971c10c564640796060fddb893


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault operator init -n 1 -t 1

Unseal Key 1: 4I9eiVZidyHnFiIxbsGLCRFAH0hsIr7QuXouuaxPMls=

Initial Root Token: hvs.TKhUKqSm0rKoi7RMFlAeOVGN

Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 1 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal 4I9eiVZidyHnFiIxbsGLCRFAH0hsIr7QuXouuaxPMls=
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.10.2
Storage Type    file
Cluster Name    vault-cluster-b290e2fe
Cluster ID      3f4b38d5-b992-8b9a-56c4-0a204f5e9a85
HA Enabled      false


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault login hvs.TKhUKqSm0rKoi7RMFlAeOVGN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.TKhUKqSm0rKoi7RMFlAeOVGN
token_accessor       wVErqZ8AVQj2ilq3QxNJu2z9
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault write -f transit/keys/autounseal
Success! Data written to: transit/keys/autounseal


root@vault0:~# tee autounseal.hcl <<EOF
> path "transit/encrypt/autounseal" {
>    capabilities = [ "update" ]
> }
>
> path "transit/decrypt/autounseal" {
>    capabilities = [ "update" ]
> }
> EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault policy write autounseal autounseal.hcl
Success! Uploaded policy: autounseal


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault token create -policy="autounseal" -wrap-ttl=600
Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIBxtJyFl_z1o5_I4yzXXBpXfhYLFvLFI53Hqju-Ay05CGh4KHGh2cy5XczJwZW9KdDNqU3hDSDNJejhhVDQ2ME8
wrapping_accessor:               wXJDdpeU063dUDMUu5vYMfCz
wrapping_token_ttl:              10m
wrapping_token_creation_time:    2022-05-10 08:52:45.232620021 +0000 UTC
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                NI1RNA7MM6JKCuR9aqAWrE9a


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 VAULT_TOKEN=hvs.CAESIBxtJyFl_z1o5_I4yzXXBpXfhYLFvLFI53Hqju-Ay05CGh4KHGh2cy5XczJwZW9KdDNqU3hDSDNJejhhVDQ2ME8 vault unwrap
Key                  Value
---                  -----
token                hvs.CAESIDq_-C47Tjl937Yn8medS3kqD24S7wXB_9DkmibXUuzRGh4KHGh2cy5qWUhlRGlSNGFBbjRwZFVpbkVYdEdVTjI
token_accessor       NI1RNA7MM6JKCuR9aqAWrE9a
token_duration       768h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]


root@vault0:~# mkdir -p vault2/config


root@vault0:~# mkdir -p vault2/data


root@vault0:~# sudo chown 100:1000 vault2/data


root@vault0:~# ip a | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.10.8/24 brd 192.168.10.255 scope global eth0


root@vault0:~# tee vault2/config/config.hcl <<EOF
> storage "file" {
>     path = "/vault/data"
> }
>
> listener "tcp" {
>   tls_disable = 1
>   address = "[::]:8200"
>   cluster_address = "[::]:8201"
> }
>
> seal "transit" {
>   address = "http://192.168.10.8:8200"
>   disable_renewal = "false"
>   key_name = "autounseal"
>   mount_path = "transit/"
>   tls_skip_verify = "true"
> }
> EOF
storage "file" {
    path = "/vault/data"
}

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}

seal "transit" {
  address = "http://192.168.10.8:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}
root@vault0:~# docker run -d --name vault2 --cap-add=IPC_LOCK -p 8100:8200 -e VAULT_TOKEN="hvs.CAESIDq_-C47Tjl937Yn8medS3kqD24S7wXB_9DkmibXUuzRGh4KHGh2cy5qWUhlRGlSNGFBbjRwZFVpbkVYdEdVTjI" -v $(pwd)/vault2/config:/vault/config -v $(pwd)/vault2/data:/vault/data vault server
b4980b7d3a73c6e55ab8d7305c72e1b3b8a835509a31e61ea767ce0e71039421


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault operator init -recovery-shares=1 -recovery-threshold=1
Recovery Key 1: Zl0qWFONNyeTa7k93SCVfFgD2ruLFC/UbEBaGOiQT98=

Initial Root Token: hvs.lOyJ0NYfcEH152iMKWZJ0TiO

Success! Vault is initialized

Recovery key initialized with 1 key shares and a key threshold of 1. Please
securely distribute the key shares printed above.


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault operator unseal Zl0qWFONNyeTa7k93SCVfFgD2ruLFC/UbEBaGOiQT98=
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.10.2
Storage Type             file
Cluster Name             vault-cluster-13cbdf12
Cluster ID               2a1dff0c-4a99-8291-72ba-a2420718dc85
HA Enabled               false


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault login hvs.lOyJ0NYfcEH152iMKWZJ0TiO
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.lOyJ0NYfcEH152iMKWZJ0TiO
token_accessor       QlUadxBcbLqoXGuUhfuseLJm
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault write -f transit/keys/autounseal
Success! Data written to: transit/keys/autounseal


root@vault0:~# tee autounseal.hcl <<EOF
> path "transit/encrypt/autounseal" {
>    capabilities = [ "update" ]
> }
>
> path "transit/decrypt/autounseal" {
>    capabilities = [ "update" ]
> }
> EOF
path "transit/encrypt/autounseal" {
   capabilities = [ "update" ]
}

path "transit/decrypt/autounseal" {
   capabilities = [ "update" ]
}
root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault policy write autounseal autounseal.hcl
Success! Uploaded policy: autounseal


root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 vault token create -policy="autounseal" -wrap-ttl=600
Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIIshRoK_oWRWk4Z8NKxKnoJ1OWIlLv_HNUZAPAgcU_uqGh4KHGh2cy5KdFlRSjZNMmxiMEQxUUFCSUppZ1dNd0M
wrapping_accessor:               TkbN42YgbfsPSouRa4bz548a
wrapping_token_ttl:              10m
wrapping_token_creation_time:    2022-05-10 08:56:30.870535933 +0000 UTC
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                e8RLO1OGHBMu9v3xHpQvx868
root@vault0:~# VAULT_ADDR=http://127.0.0.1:8100 VAULT_TOKEN=hvs.CAESIIshRoK_oWRWk4Z8NKxKnoJ1OWIlLv_HNUZAPAgcU_uqGh4KHGh2cy5KdFlRSjZNMmxiMEQxUUFCSUppZ1dNd0M vault unwrap
Key                  Value
---                  -----
token                hvs.CAESIPnAHrQ2EOCCukVxwEIAIv07QZCvx9tmpYB7cy0bNsWRGh4KHGh2cy5jT0JTbUc0Wk9pQ1JZTEw4TVVOenhEVHg
token_accessor       e8RLO1OGHBMu9v3xHpQvx868
token_duration       768h
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]


root@vault0:~# ip a | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.10.8/24 brd 192.168.10.255 scope global eth0


root@vault0:~# tee -a vault/config/config.hcl <<EOF
>
> seal "transit" {
>   address = "http://192.168.10.8:8200"
>   disable_renewal = "false"
>   key_name = "autounseal"
>   mount_path = "transit/"
>   tls_skip_verify = "true"
> }
> EOF

seal "transit" {
  address = "http://192.168.10.8:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  tls_skip_verify = "true"
}
root@vault0:~# docker rm -f vault
vault

root@vault0:~# docker run -d --name vault --cap-add=IPC_LOCK -p 8200:8200 -e VAULT_TOKEN="hvs.CAESIPnAHrQ2EOCCukVxwEIAIv07QZCvx9tmpYB7cy0bNsWRGh4KHGh2cy5jT0JTbUc0Wk9pQ1JZTEw4TVVOenhEVHg" -v $(pwd)/vault/config:/vault/config -v $(pwd)/vault/data:/vault/data vault server
dc754be8c0239466822e6215bfe6d23225d4a7cf8e2282cb04f21759d4ae099d

root@vault0:~# VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal 4I9eiVZidyHnFiIxbsGLCRFAH0hsIr7QuXouuaxPMls=
Error unsealing: Put "http://127.0.0.1:8200/v1/sys/unseal": dial tcp 127.0.0.1:8200: connect: connection refused

root@vault0:~# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS                      PORTS                                       NAMES
dc754be8c023   vault     "docker-entrypoint.s…"   37 seconds ago   Exited (1) 15 seconds ago                                               vault
b4980b7d3a73   vault     "docker-entrypoint.s…"   4 minutes ago    Up 4 minutes                0.0.0.0:8100->8200/tcp, :::8100->8200/tcp   vault2

root@vault0:~# docker logs dc754be8c023
2022-05-10T08:58:30.734Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2022-05-10T08:58:34.373Z [INFO]  seal.transit: unable to renew token, disabling renewal: err="Put \"http://192.168.10.8:8200/v1/auth/token/renew-self\": EOF"
Error parsing Seal configuration: Put "http://192.168.10.8:8200/v1/transit/encrypt/autounseal": EOF
root@vault0:~#

image

Hmm. There may be an error in the configuration. Because the fallen container itself listens to 8200

@patsevanton
Copy link

Corrected in the config 'vault/config/config.hcl`:

seal "transit" {
  address = "http://192.168.10.8:8100"
}

192.168.10.8:8100 - second vault.
Now another issue:

VAULT_ADDR=http://127.0.0.1:8200 vault operator unseal 4I9eiVZidyHnFiIxbsGLCRFAH0hsIr7QuXouuaxPMls=
Error unsealing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:

* migrate option not provided and seal migration is pending

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment