Skip to content

Instantly share code, notes, and snippets.

@voron
Last active July 31, 2020 11:55
Show Gist options
  • Select an option

  • Save voron/6954a6af3a193ad2ae7ba1d34bbf9b00 to your computer and use it in GitHub Desktop.

Select an option

Save voron/6954a6af3a193ad2ae7ba1d34bbf9b00 to your computer and use it in GitHub Desktop.
helm2 statefulset upgrade
I have a helm release with statefulset, services and configmaps
-------------------------------------------------------------------
kubectl get cm,service,statefulset -o yaml|egrep 'kind|uid|creationTimestamp'|grep -v null
kind: ConfigMap
creationTimestamp: "2020-07-06T18:36:05Z"
uid: 20acf814-d3df-49ca-830d-94e00f54af09
kind: ConfigMap
creationTimestamp: "2020-07-06T18:36:05Z"
uid: b166cbd3-b31d-4736-87b1-f31a103784fe
kind: Service
creationTimestamp: "2020-07-07T18:56:17Z"
uid: fd0b50ec-d163-4bf8-a1aa-e114f2f72209
kind: Service
creationTimestamp: "2020-07-06T18:36:05Z"
uid: 5478524e-e5ba-4fab-ba6c-486fca8d7235
kind: StatefulSet
creationTimestamp: "2020-07-06T19:23:27Z"
uid: 7a569ba1-9104-487d-aabf-d59d0682d6c5
kind: List
-------------------------------------------------------------------
I change storage size in volumeClaimTemplates inside StatefulSet via helm values.
And I try to upgrade the release via helm2 upgrade w/o `--force` and get an expected error:
===================================================================
UPGRADE FAILED
Error: StatefulSet.apps "dev-bsc0" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
ROLLING BACK Rollback was a success.
===================================================================
K8s resources are not modified, uid and creationTimestamp are the same.
Now I add `--force` to helm2 upgrade
===================================================================
Release "dev-bsc0" has been upgraded.
LAST DEPLOYED: Fri Jul 31 12:48:51 2020
===================================================================
I check resources versions again
-------------------------------------------------------------------
kubectl get cm,service,statefulset -o yaml|egrep 'kind|uid|creationTimestamp'|grep -v null
kind: ConfigMap
creationTimestamp: "2020-07-06T18:36:05Z"
uid: 20acf814-d3df-49ca-830d-94e00f54af09
kind: ConfigMap
creationTimestamp: "2020-07-06T18:36:05Z"
uid: b166cbd3-b31d-4736-87b1-f31a103784fe
kind: Service
creationTimestamp: "2020-07-07T18:56:17Z"
uid: fd0b50ec-d163-4bf8-a1aa-e114f2f72209
kind: Service
creationTimestamp: "2020-07-06T18:36:05Z"
uid: 5478524e-e5ba-4fab-ba6c-486fca8d7235
kind: StatefulSet
creationTimestamp: "2020-07-31T09:48:51Z"
uid: 06ef5a5a-87a6-49bd-a5d5-4f6fb2ee99c9
kind: List
-------------------------------------------------------------------
And I see the only re-created resource is StatefulSet.
All other resources are not affected despite of helm upgrade --force
Here are corresponding tiller logs:
-------------------------------------------------------------------
[kube] 2020/07/31 09:48:51 building resources from updated manifest
[kube] 2020/07/31 09:48:51 checking 5 resources for changes
[kube] 2020/07/31 09:48:51 Looks like there are no changes for ConfigMap "dev-bsc0-config"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for ConfigMap "dev-bsc0-scripts"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for Service "dev-bsc0-ilb"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for Service "dev-bsc0-service"
2020/07/31 09:48:51 Cannot patch StatefulSet: "dev-bsc0" (StatefulSet.apps "dev-bsc0" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateSt
rategy' are forbidden)
2020/07/31 09:48:51 Deleted StatefulSet: "dev-bsc0"
2020/07/31 09:48:51 Created a new StatefulSet called "dev-bsc0"
[kube] 2020/07/31 09:48:51 beginning wait for 5 resources with timeout of 10m0s
-------------------------------------------------------------------
@voron

voron commented Jul 31, 2020

Copy link
Copy Markdown
Author

Here is full tiller log regarding this upgrade

[tiller] 2020/07/31 09:48:51 preparing update for dev-bsc0
[storage] 2020/07/31 09:48:51 getting deployed releases from "dev-bsc0" history
[tiller] 2020/07/31 09:48:51 resetting values to the chart's original version
[storage] 2020/07/31 09:48:51 getting last revision of "dev-bsc0"
[storage] 2020/07/31 09:48:51 getting release history for "dev-bsc0"
[tiller] 2020/07/31 09:48:51 rendering bsc chart using values
2020/07/31 09:48:51 info: manifest "bsc/templates/lb.yaml" is empty. Skipping.
2020/07/31 09:48:51 info: manifest "bsc/templates/lb-p2p.yaml" is empty. Skipping.
2020/07/31 09:48:51 info: manifest "bsc/templates/lb-p2p-discovery.yaml" is empty. Skipping.
2020/07/31 09:48:51 info: manifest "bsc/templates/ingress.yaml" is empty. Skipping.
[tiller] 2020/07/31 09:48:51 creating updated release for dev-bsc0
[storage] 2020/07/31 09:48:51 creating release "dev-bsc0.v21"
[tiller] 2020/07/31 09:48:51 performing update for dev-bsc0
[tiller] 2020/07/31 09:48:51 executing 0 pre-upgrade hooks for dev-bsc0
[tiller] 2020/07/31 09:48:51 hooks complete for pre-upgrade dev-bsc0
[kube] 2020/07/31 09:48:51 building resources from updated manifest
[kube] 2020/07/31 09:48:51 checking 5 resources for changes
[kube] 2020/07/31 09:48:51 Looks like there are no changes for ConfigMap "dev-bsc0-config"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for ConfigMap "dev-bsc0-scripts"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for Service "dev-bsc0-ilb"
[kube] 2020/07/31 09:48:51 Looks like there are no changes for Service "dev-bsc0-service"
2020/07/31 09:48:51 Cannot patch StatefulSet: "dev-bsc0" (StatefulSet.apps "dev-bsc0" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateSt
rategy' are forbidden)
2020/07/31 09:48:51 Deleted StatefulSet: "dev-bsc0"
2020/07/31 09:48:51 Created a new StatefulSet called "dev-bsc0"
[kube] 2020/07/31 09:48:51 beginning wait for 5 resources with timeout of 10m0s
[tiller] 2020/07/31 09:48:53 executing 0 post-upgrade hooks for dev-bsc0
[tiller] 2020/07/31 09:48:53 hooks complete for post-upgrade dev-bsc0
[storage] 2020/07/31 09:48:53 updating release "dev-bsc0.v20"
[tiller] 2020/07/31 09:48:53 updating status for updated release for dev-bsc0
[storage] 2020/07/31 09:48:53 updating release "dev-bsc0.v21"
[storage] 2020/07/31 09:48:54 getting last revision of "dev-bsc0"
[storage] 2020/07/31 09:48:54 getting release history for "dev-bsc0"
[kube] 2020/07/31 09:48:54 Doing get for ConfigMap: "dev-bsc0-scripts"
[kube] 2020/07/31 09:48:54 get relation pod of object: dev-bsc/ConfigMap/dev-bsc0-scripts
[kube] 2020/07/31 09:48:54 Doing get for ConfigMap: "dev-bsc0-config"
[kube] 2020/07/31 09:48:54 get relation pod of object: dev-bsc/ConfigMap/dev-bsc0-config
[kube] 2020/07/31 09:48:54 Doing get for Service: "dev-bsc0-service"
[kube] 2020/07/31 09:48:54 get relation pod of object: dev-bsc/Service/dev-bsc0-service
[kube] 2020/07/31 09:48:54 Doing get for Service: "dev-bsc0-ilb"
[kube] 2020/07/31 09:48:54 get relation pod of object: dev-bsc/Service/dev-bsc0-ilb
[kube] 2020/07/31 09:48:54 Doing get for StatefulSet: "dev-bsc0"
[kube] 2020/07/31 09:48:54 get relation pod of object: dev-bsc/StatefulSet/dev-bsc0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment