Skip to content

Instantly share code, notes, and snippets.

@mgoodness
Last active September 7, 2024 16:25
Show Gist options
  • Save mgoodness/1a2926f3b02d8e8149c224d25cc57dc1 to your computer and use it in GitHub Desktop.
Save mgoodness/1a2926f3b02d8e8149c224d25cc57dc1 to your computer and use it in GitHub Desktop.
AWS ELB-related annotations for Kubernetes Services (as of v1.12.0)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval (in minutes)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix
  • service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags (comma-separated list of key=value)
  • service.beta.kubernetes.io/aws-load-balancer-backend-protocol (http|https|ssl|tcp)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout (in seconds)
  • service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout (in seconds, default 60)
  • service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-extra-security-groups (comma-separated list)
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-internal (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert (IAM or ACM ARN)
  • service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports (default '*')
  • service.beta.kubernetes.io/aws-load-balancer-type: nlb
@guitarrapc
Copy link

@gangstead
Copy link

Regarding multiple ssl certs on NLB. Current k8s service only supports a single ARN for a certificate https://kubernetes.io/docs/concepts/services-networking/service/#ssl-support-on-aws
BUT the aws-load-balancer-controller takes a stringList of certificate ARNs allowing you to use multiple certificates.

I had a really hard time figuring out that those aws-load-balancer* annotations are standard k8s but if you install the aws-load-balancer-controller then it replaces how service resources are handled and it almost handles the same annotations in the same way. The important difference for me is the handling of multiple certificates in the service.beta.kubernetes.io/aws-load-balancer-ssl-cert annotation now becomes possible.

@RichardoC
Copy link

RichardoC commented Mar 14, 2022

Hi All,
Do these following health check annotations work with AWS NLB?
service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
For us, its not reflecting changes in console once we apply them.
Or do we have any specific list of annotations that works well with NLB.
--Tarun

Same for me, when will this be fixed? Thanks.

Did anyone find/open a bug for this? I'm encountering the same on EKS 1.19

Found kubernetes/kubernetes#94546 which fixes the health check issues I was encountering

@Ghilteras
Copy link

I dont think service.beta.kubernetes.io/aws-load-balancer-type supports nlb
it has to be nlb-ip

@haarchri
Copy link

is private static IPs supportet via annotations ?

@c4m4
Copy link

c4m4 commented Jun 27, 2022

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

For backwards compatibility, controller still supports the nlb-ip as the type annotation. For example, if you specify

service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip

@c4m4
Copy link

c4m4 commented Jun 27, 2022

is private static IPs supportet via annotations ?

To use an static ip, you need to specify this in the k8s service

@haarchri
Copy link

haarchri commented Jun 27, 2022

we have set the following annotations to our ingress-controller to automatically spawn an nlb in front of our eks/ingress-nginx

kubectl describe service ingress-nginx-controller -n ingress-nginx:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-internal: true
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.19.64.10, 10.19.128.10, 10.19.192.10
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb

But the nlb always spawns with dynamic IPs.

@RichardoC
Copy link

we have set the following annotations to our ingress-controller to automatically spawn an nlb in front of our eks/ingress-nginx

kubectl describe service ingress-nginx-controller -n ingress-nginx:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-internal: true
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.19.64.10, 10.19.128.10, 10.19.192.10
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb

But the nlb always spawns with dynamic IPs.

Are you using the AWS Load Balancer Controller or what's built in to EKS? That annotation only seems to be used by the AWS LBC which needs to be installed separately.
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/

@kamaraj-muthupandian
Copy link

is http2 supported via annotations ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment