Skip to content

Instantly share code, notes, and snippets.

@alena1108
Created October 31, 2019 21:20
Show Gist options
  • Save alena1108/3b1e4b052646ecfff909b81c0bd6a3f9 to your computer and use it in GitHub Desktop.
Save alena1108/3b1e4b052646ecfff909b81c0bd6a3f9 to your computer and use it in GitHub Desktop.
**What kind of request is this (question/bug/enhancement/feature request):** bug
**Steps to reproduce (least amount of steps as possible):**
- Enable custom config in an exiting rke cluster.
- Edit cluster, Add this in the YAML file
```
services:
kube-api:
secrets_encryption_config:
enabled: true
custom_config:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: k-jgxw9
secret: QkNEQTRGRUMyMTExOUM4M0FBNjY4MTlDNTBDQTY4RkE=
- identity: {}
```
- On clicking Save, cluster is provisioned again, goes into Updating state and comes up Active.
- The secrets are seen encrypted on the etcd
```
/ # ETCDCTL_API=3 etcdctl get /registry/secrets/default/test1 |hexdump -C
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 74 65 73 74 31 0a |s/default/test1.|
00000020 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 3a 76 |k8s:enc:aescbc:v|
00000030 31 3a 6b 2d 6a 67 78 77 39 3a 4d f0 d2 16 f4 3f |1:k-jgxw9:M....?|
00000040 c5 3e aa 4d b5 de 08 0f da 06 02 40 d0 ef 5e 0b |.>.M.......@..^.|
00000050 e6 ba df 24 44 1e 17 13 75 86 fe a2 d2 83 6e e1 |...$D...u.....n.|
00000060 7d da 3c 77 1b 20 68 68 ab dd f0 82 3e b8 f9 42 |}.<w. hh....>..B|
00000070 a1 60 93 49 1f 0f 95 90 ba 26 d6 bf d4 2b 16 41 |.`.I.....&...+.A|
00000080 ff df 6a 88 a8 8d 99 29 45 30 72 f8 3b 81 fe 7d |..j....)E0r.;..}|
00000090 46 f1 47 85 27 86 7b aa 38 36 36 70 5e 07 c2 e6 |F.G.'.{.866p^...|
000000a0 f3 17 a0 14 c2 2a 78 88 cf b9 78 61 7f 05 78 e2 |.....*x...xa..x.|
000000b0 73 46 a1 e0 c3 eb 35 72 da b0 6d e9 cc 3f ae 50 |sF....5r..m..?.P|
000000c0 6c 45 46 e4 82 d7 2a ba 7e 3f a0 d6 b0 75 b8 b0 |lEF...*.~?...u..|
000000d0 6c 04 ef a3 24 23 3a 9a 67 c5 51 60 c4 60 55 81 |l...$#:.g.Q`.`U.|
000000e0 61 4f 07 34 97 8f c6 03 45 0f 05 df a4 3e 11 74 |aO.4....E....>.t|
000000f0 0f 1f cf 36 27 dd 1d 3f 07 06 68 01 24 b9 bd 6e |...6'..?..h.$..n|
00000100 7d 55 7c 7b 3e 24 ef 72 38 51 8c a2 03 33 ed af |}U|{>$.r8Q...3..|
00000110 94 bb 02 03 59 56 87 80 97 fc 4a 07 39 d8 8a c2 |....YV....J.9...|
00000120 1b 44 5f e8 e6 25 3b 44 0c c7 4b 87 d0 d9 c8 22 |.D_..%;D..K...."|
00000130 55 18 f6 bf f1 46 70 19 de de 2b e3 bb ac 44 4c |U....Fp...+...DL|
00000140 5e 6b 46 ec 48 78 18 fc 6f 20 71 7b f7 f4 e3 04 |^kF.Hx..o q{....|
00000150 ab 34 79 c9 97 37 33 43 3d c2 7c bf 72 34 1d 1e |.4y..73C=.|.r4..|
00000160 95 aa 1f 64 7b 22 a3 e6 c6 da 38 70 35 5f 05 70 |...d{"....8p5_.p|
00000170 a8 f5 ab 38 92 cf b5 92 d9 f9 3b db 71 0d 24 8a |...8......;.q.$.|
00000180 a8 01 34 87 68 f7 ba 6e e9 4c 0a |..4.h..n.L.|
```
- `encryption.yaml` file on the control plane node is
```
/etc/kubernetes/ssl# cat encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- providers:
- aescbc:
keys:
- name: k-jgxw9
secret: QkNEQTRGRUMyMTExOUM4M0FBNjY4MTlDNTBDQTY4RkE=
- identity: {}
resources:
- secrets
```
- Now add new key to the config, in the first place in the keys list. Old key will be second in the list
- `encryption.yaml` file:
```
root@santest115c1:/etc/kubernetes/ssl# cat encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- providers:
- aescbc:
keys:
- name: k-abcde
secret: AbCDETRGRUMyMTExOUM4M0FBNjY4MTlDNTBDQTY4RkE=
- name: k-jgxw9
secret: QkNEQTRGRUMyMTExOUM4M0FBNjY4MTlDNTBDQTY4RkE=
- identity: {}
resources:
- secrets
```
- The secrets continue to be encrypted with **`k-jgxw9`** key
- Remove the old key from the config. Click on Save
- After the cluster comes up active, `encryption.yaml` file:
```
root@santest115c1:/etc/kubernetes/ssl# cat encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- providers:
- aescbc:
keys:
- name: k-abcde
secret: AbCDETRGRUMyMTExOUM4M0FBNjY4MTlDNTBDQTY4RkE=
- identity: {}
resources:
- secrets
```
- But the secrets continue to be encrypted with **`k-jgxw9`** key
**Environment information**
- Rancher version (`rancher/rancher`/`rancher/server` image tag or shown bottom left in the UI): 2.3-head latest
- Installation option (single install/HA): single
<!--
If the reported issue is regarding a created cluster, please provide requested info below
-->
**Cluster information**
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported): rke DO
- Kubernetes version (use `kubectl version`):
```
1.15.5
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment