Background Context:
- Question: What is a helm chart?
- Answer: helm charts are just templating engines that render generated yaml
- input variables (with a mix of default and override values for those variables)
- get plugged into golang templatized yaml
- and output a wall of text of generated yaml
- Community Helm Charts:
- will often allow customizability of commonly edited features (like replicaCount)
- will rarely allow customizability of relatively new, rarely used, and niche Kubernetes features; such as: runtime classes, priority classes, prestop lifecycle hooks, and securityContext
- this is done because someone has to maintain the yaml generating templates, lower customizable options = easier maintainability.
- Common Use Case - where using kustomize to do a last mile customization of a helm chart is useful:
Let's say we need to customize the securityContext of a pod, (or implement any other rarely used setting), and the helm chart doesn't support that customization. - In the past you had the option of "forking" a helm chart, which everyone hated
- Forking is git speak for copy. Here's what forking involves:
- You basically create a copy of the helm chart's templatization source files, customize them to your needs, and store them in a git repo.
- Decide if your org's internal users will all install/reference the helm chart directly the git repo (a great simple option), or if you want to go down a devops yak shaving rabbit hole to create helm repo and upload your customize helm chart to your helm repo. (Most Community helm charts are hosted on helm repos, where the templatized files are abstracted away from end users.)
- Decide how to inform your colleagues that your app was installed with a custom helm chart and not the community helm chart they found via google. (great time to document for internal users things like what chart is used, where it's sourced from, any one off pre-req commands needed to use it, naming convention to use for release name, what namespace to put it in, an example of what the install command is supposed to look like.)
- Decide if you will maintain your customized fork of the helm chart
- No: means you won't be able to benefit from community improvements to the helm chart over time
- Yes: means you've created busy work for yourself (cuppa tech debt anyone?)
- The unlucky among you may cut their teeth discovering the annoying gotcha of helm customizations:
somechart.helm-values.yamlrepresents value overrides of the default values of variables defined in the chart'svalues.yamlfile.- yaml lists in
somechart.helm-values.yamlwillREPLACEyaml lists invalues.yaml - yaml hashmaps
somechart.helm-values.yamlwillMERGEwith yaml hashmaps invalues.yaml - In most cases merge is the desired behavior, replace results in unintuitive debugging, because you'd expect consistency, so inless you know what you're looking for debugging can be rough.
- The common workaround is to heavily perfer hashmaps, and jump through all kinds of crazy hoops to use hashmaps, including some advanced list to hashmap conversion magic. All of which tends to add complexity to a helm chart's templatized yaml files.
- If you customize a helm chart, you might suddenly get exposed to crazy complex workarounds that can the chart work really well from an end user experience, but become very complex on the helm chart maintainer side, due to the complex workarounds making things unreadable which results in code that's hard to maintain and customize.
- Forking is git speak for copy. Here's what forking involves:
Answer to "Why would you want to do this?"
- The point of why using kustomize as a helm chart post renderer can make sense:
- Using kustomize to do last mile customizations of a helm chart is an alternative to forking helm charts in order to implement customizations not supported by a community helm chart.
(In some limited cases kustomize can end up being less work, easier to maintain, and in most cases allows you to easily consume improvements/updates to the community helm chart, with zero modifications of the kustomize yaml patch file.) - Note: This is also a bit hacky/fragile.
- Scenarios where it tends to be a good fit:
- 1 off fix where you need to capture something as IaC, the deployment is practically E2E automated by a pipeline, and and you don't plan to change much. (such as always deploy using the same helm release name and deploy to the same namespace.)
- Kustomize yaml patch files sometimes introduce tight coupling with namespaces and how objects are named, and helm release names often influence things like deployment names. (If you need a solution that works for a helm chart that's always deployed with a consistent name, this can be good.)
- If you need a more robust solution that's less fragile (like won't break if a helm release is renamed or deployed in a different namespace) and works for more general cases then forking helm will likely be the better option.
- You may be able to workaround it if the apply command is wrapped in a deployment pipeline that allows helm, kustomize, and the files they use to be fully templatized. (Think Ansible/Jinja2, in such a case the helm release name and namespace could also be templatized.)
- Scenarios where it tends to be a good fit:
- Using kustomize to do last mile customizations of a helm chart is an alternative to forking helm charts in order to implement customizations not supported by a community helm chart.
- You have access to a bash/zsh terminal
- helm and kustomize cli are installed
(tested with helm v3.11.2 & kustomize v4.5.7, but any 3.x & 4.x versions should work) - I'll assume that you'll copy paste the majority of the shell commands 1 line at a time, and ignore the comments/not copy paste the grey comments that start with #.
- I'll assume you'll copy paste the multi-line-commands as multi-line-commands
- when selecting the text for copy
- start from
cat << EOF | tee ~/file-location - and end at the next instance of
EOF
- The following helm chart is intended to be a mock representation of a community helm chart
# [admin@bash_shell:~]
# We'll use the built in example helm chart for this exercise
mkdir -p ~/example
cd ~/example
# [admin@bash_shell:~/example]
helm create myhelmchart
# ^-- the above command created a folder with pre-populated contents, the folder represents a helm chart
cat ~/example/myhelmchart/values.yaml
# ^-- This file represents the helm chart's defined variables & their default values
cat ~/example/myhelmchart/values.yaml | grep replicaCount
# replicaCount: 1
# ^-- Notice that the defined variable "replicaCount" has a default value of 1# v-- dry run command (with debug flag that tells you input parameters)
helm upgrade --install myhelmrelease ~/example/myhelmchart --namespace=default --debug --dry-run
# v-- live apply command
helm upgrade --install myhelmrelease ~/example/myhelmchart --namespace=default
# v-- feedback command
kubectl get podsShell Output:
NAME READY STATUS RESTARTS AGE
myhelmrelease-myhelmchart-5ffd9df444-hf5b8 1/1 Running 0 3s- This file should be a near copy or subset of the default values.yaml, any values you specify here will override the default values.
- We're doing this so our example command will be inline with how community helm charts are normally used
# [admin@bash_shell:~/example]
# Copy paste the following multi-line-command to create a file with pre-populated contents
cat << EOF | tee ~/example/myhelmchart.helm-values.yaml
replicaCount: 2
EOF# v-- dry run command (with debug flag that tells you input parameters)
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --debug --dry-run
# v-- live apply command
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default
# v-- feedback command
kubectl get podsShell Output:
NAME READY STATUS RESTARTS AGE
myhelmrelease-myhelmchart-5ffd9df444-sjkv8 1/1 Running 0 44s
myhelmrelease-myhelmchart-5ffd9df444-nxrng 1/1 Running 0 11s- myhelmchart is a mock representation of a community helm chart
(so we'll pretend it's not editable & that we don't want to fork it) - Our goal will be to customize the workload to include the following rarely used configurations:
- we want to add
priorityClassName: system-cluster-criticalto the pod spec - we want to add postStart and preStop lifecycle hooks to a container in the pod spec
- we want to add
- The following command proves our helm chart lacks out of the box support for either customization:
# [admin@bash_shell:~/example]
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
-n=default --debug --dry-run 2>&1 | grep -B 10000 HOOKS | grep -v HOOKS | grep -A 10000 USER-SUPPLIEDShell Output:
USER-SUPPLIED VALUES:
replicaCount: 2
COMPUTED VALUES:
affinity: {}
autoscaling:
enabled: false
maxReplicas: 100
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
pullPolicy: IfNotPresent
repository: nginx
tag: ""
imagePullSecrets: []
ingress:
annotations: {}
className: ""
enabled: false
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 2
resources: {}
securityContext: {}
service:
port: 80
type: ClusterIP
serviceAccount:
annotations: {}
create: true
name: ""
tolerations: []Step 3: Now we'll use kustomize cli to do last mile customization of the helm chart using helm's post-rendering feature
- This is basically an example of what's mentioned in the relevant helm docs
- we'll use a multi-line-copy pasteable command to generate a bash script
- The scripts actually logic is 2 lines long (the extra lines are following best practice of referencing script intepreter & documentation)
- Then we'll make it executable
# [admin@bash_shell:~/example]
# v-- generating a custom helm post renderer script
cat << EOF | tee ~/example/kustomize.sh
#!/bin/bash
######################################################
# Usage Notes:
# This is a utility script and it'll be triggered /
# called by the helm CLI, when helm's --post-renderer
# flag is used and references this script
#
# helm ...(will output generated yaml)... | kustomize.sh
# ^-- Imagine that the yaml wall of text generated by helm
# will become stdin for this script
# the contents of stdin will be put in a temp file
# (because `kustomize build .` will expect a file to exist)
# Then the temp file will be removed after being
# used by kustomize for post rendering purposes
#
# remember `-` is bash representation of stdin
######################################################
cat - > helm-generated-output.yaml
kustomize build . && rm helm-generated-output.yaml
EOF
sudo chmod +x ~/example/kustomize.sh
# ^-- making the generated file executable- kustomize.sh will run
kustomize build .(which will use this file) - we'll use a multi-line-copy pasteable command to generate a bash script
# [admin@bash_shell:~/example]
cat << EOF | tee ~/example/kustomization.yaml
resources:
- helm-generated-output.yaml
patchesStrategicMerge:
- helm-post-rendering-patch.yaml
EOF-
Run the following command to get an idea of what files helm is generating
# [admin@bash_shell:~/example] helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \ --namespace=default --dry-run | grep "# Source:"
Shell Output:
# Source: myhelmchart/templates/tests/test-connection.yaml # Source: myhelmchart/templates/serviceaccount.yaml # Source: myhelmchart/templates/service.yaml # Source: myhelmchart/templates/deployment.yaml
-
Run the following command to take a closer look at an individual generated file.
Notes:
1: This deployment.yaml represents the object we want to do some post rendering customimization of using kustomize
2: notice that templates/deployment.yaml came from the above command (this could be a different file in a different helm chart)# [admin@bash_shell:~/example] helm template --show-only templates/deployment.yaml --namespace=default \ myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml
Shell Output:
--- # Source: myhelmchart/templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: myhelmrelease-myhelmchart labels: helm.sh/chart: myhelmchart-0.1.0 app.kubernetes.io/name: myhelmchart app.kubernetes.io/instance: myhelmrelease app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm spec: replicas: 2 selector: matchLabels: app.kubernetes.io/name: myhelmchart app.kubernetes.io/instance: myhelmrelease template: metadata: labels: app.kubernetes.io/name: myhelmchart app.kubernetes.io/instance: myhelmrelease spec: serviceAccountName: myhelmrelease-myhelmchart securityContext: {} containers: - name: myhelmchart securityContext: {} image: "nginx:1.16.0" imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 protocol: TCP livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: / port: http resources: {}
- The name is semi arbitrary as long as it's consistent with whats in kustomization.yaml
- The contents of the yaml patch file, should be a partial match fo some of the YAML generated by helm's output, the partial matches are used to figure out what needs to be merged. Then the differences are overridden with the patch taking priority.
- We'll use a multi-line-copy pasteable command to generate a bash script
# [admin@bash_shell:~/example]
cat << EOF | tee ~/example/helm-post-rendering-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myhelmrelease-myhelmchart # <-- weakness of this approach is tight coupling to helm release / chart name
namespace: default # <-- weakness of this approach is tight coupling to namespace
spec:
template:
spec:
priorityClassName: system-cluster-critical
containers:
- name: myhelmchart #<-- this is chart specific
securityContext: null #<-- side note: you can set a value to null to trigger removal of values
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop:
exec:
command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
EOF- helm cli will trigger
- kustomize.sh (which generates a file) then triggers kustomize cli
- kustomize cli looks for kustomization.yaml for instructions
- kustomization.yaml's logic is dead simple:
- strategic patch merge of complete yaml wall of text provided by helm
- with partial yaml patch file
# [admin@bash_shell:~/example]
# v-- example syntax of how to run command in dryrun mode
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --post-renderer ./kustomize.sh --debug --dry-run
# v-- example syntax of how to run command
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --post-renderer ./kustomize.sh# [admin@bash_shell:~/example]
# v-- dry run command with grep to verify change of adding priorityClassName
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --dry-run | grep priorityClassName
# ^-- expected output is blank
# v-- dry run command with grep to verify change
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --post-renderer ./kustomize.sh --dry-run | grep priorityClassName
# ^-- expected output is priorityClassName: system-cluster-critical
# v-- dry run command with grep to verify change of adding priorityClassName
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --dry-run | grep preStop:
# ^-- expected output is blank
# v-- dry run command with grep to verify change
helm upgrade --install myhelmrelease ~/example/myhelmchart -f ~/example/myhelmchart.helm-values.yaml \
--namespace=default --post-renderer ./kustomize.sh --dry-run | grep -A 12 lifecycle:
# ^-- expected output is 13 line snippet of yaml config about lifecycle hooks we injected earlier