Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
AWS ELB-related annotations for Kubernetes Services (as of v1.12.0)

AWS Service annotations

  • service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval (in minutes)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix
  • service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags (comma-separated list of key=value)
  • service.beta.kubernetes.io/aws-load-balancer-backend-protocol (http|https|ssl|tcp)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout (in seconds)
  • service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout (in seconds, default 60)
  • service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-extra-security-groups (comma-separated list)
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-internal (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert (IAM or ACM ARN)
  • service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports (default '*')
  • service.beta.kubernetes.io/aws-load-balancer-type: nlb
@michaeljmarshall
Copy link

michaeljmarshall commented Jun 2, 2020

AWS support actually recommends to comment out the annotation during creation, then remove the comment and update the service.

@kostas-rs - any chance you can provide a reference to where AWS support recommends the two stage creation? Thanks!

@kostas-rs
Copy link

kostas-rs commented Jun 2, 2020

AWS support actually recommends to comment out the annotation during creation, then remove the comment and update the service.

@kostas-rs - any chance you can provide a reference to where AWS support recommends the two stage creation? Thanks!

@michaeljmarshall I don't have a public reference but when i reached out to them via support they provided this as a workaround.

@prashantbhatiain
Copy link

prashantbhatiain commented Jun 28, 2020

Since Multiple SSL certificates are supported on NLB ,is there any annotation to support that .For example , i was trying below configuration for one of my ingress controllers but this doesn't seem to work .However ,i'm able to add multiple certificates from AWS console .

service.beta.kubernetes.io/aws-load-balancer-ssl-cert : arn:aws:acm:us-east-1:XXXXXXXXX:certificate/AAAAAAAAA,arn:aws:acm:us-east-1:XXXXXXXXXX:certificate/BBBBBBBB

@max-lobur
Copy link

max-lobur commented Aug 17, 2020

Is there any way to limit nodes that being added to the resulting loadbalancer? E.g. I have a few workers groups, with different labels, and want ELB to include only 1 of them in backends.

@denniswebb
Copy link

denniswebb commented Aug 18, 2020

Change spec.externalTrafficPolicy to Local on your service spec. @max-lobur

@sgasquet
Copy link

sgasquet commented Sep 10, 2020

Since Multiple SSL certificates are supported on NLB ,is there any annotation to support that .For example , i was trying below configuration for one of my ingress controllers but this doesn't seem to work .However ,i'm able to add multiple certificates from AWS console .

service.beta.kubernetes.io/aws-load-balancer-ssl-cert : arn:aws:acm:us-east-1:XXXXXXXXX:certificate/AAAAAAAAA,arn:aws:acm:us-east-1:XXXXXXXXXX:certificate/BBBBBBBB

Hi, i'm interested too in such possibility. Is there anyone facing this same need to have multiple TLS terminations on a same NLB Loadbalancer ?

@SrikanthBhandary
Copy link

SrikanthBhandary commented Sep 28, 2020

Is there any way to specify the target type as ip based?

@guitarrapc
Copy link

guitarrapc commented Sep 29, 2020

alb-ingress-controller v2 will offer NLB with target type ip.
Currently it's rc1 so not for production use.

https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v2.0.0-rc1

@c4m4
Copy link

c4m4 commented Oct 1, 2020

@c4m4, did you figure that one out? I'm also looking to select a specific subnet per load balancer.

I solved using eks, but with kops is the same, using this tag on the subnet

"kubernetes.io/cluster/${local.cluster_name}" = "shared"

@gerardgorrion
Copy link

gerardgorrion commented Oct 26, 2020

Hi all, is possible to add tags into yaml file to deploy NLB? We use service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags but no showed into aws console... Thanks!

@japzio
Copy link

japzio commented Oct 26, 2020

@malcolm061990
Copy link

malcolm061990 commented Oct 27, 2020

Hi, guys. In AWS EKS I have two NLB's.
And I have two ingresses for different hostnames:
domain1.com
domain2.com

I want ingress for domain1.com using only first NLB and ingress for domain2.com using only second NLB. I can't understand how can I configure ingress to use specified NLB.

@c4m4
Copy link

c4m4 commented Oct 27, 2020

do you have two ingress rules or two ingress controller? you could have one ingress controller that manager all the ingress rules and use only one lb

@malcolm061990
Copy link

malcolm061990 commented Oct 27, 2020

do you have two ingress rules or two ingress controller? you could have one ingress controller that manager all the ingress rules and use only one lb

I have two ingress rules in different ns and one ingress controller. But I have two nlb, each with its own certificate from AWS cert manager

@c4m4
Copy link

c4m4 commented Oct 27, 2020

Which kind of ingress controller you are using?

@malcolm061990
Copy link

malcolm061990 commented Oct 27, 2020

Which kind of ingress controller you are using?

Nginx ingress controller

@malcolm061990
Copy link

malcolm061990 commented Nov 2, 2020

I fixed my issue using kubernetes.io/ingress.class: INGRESS_APPLICATION annotation in ingress resource, --ingress-class=%INGRESS_APPLICATION% arg and --publish-service=$(POD_NAMESPACE)/ingress-nginx-%INGRESS_APPLICATION% arg in nginx ingress controller.

@johnzheng1975
Copy link

johnzheng1975 commented Nov 16, 2020

Hi All,

Do these following health check annotations work with AWS NLB?

service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold

For us, its not reflecting changes in console once we apply them.

Or do we have any specific list of annotations that works well with NLB.

--Tarun

Same for me, when will this be fixed? Thanks.

@adamdonahue
Copy link

adamdonahue commented Jan 5, 2021

Any update on the possibility of naming the load balancer via an annotation?

@guitarrapc
Copy link

guitarrapc commented Jan 6, 2021

@gangstead
Copy link

gangstead commented May 14, 2021

Regarding multiple ssl certs on NLB. Current k8s service only supports a single ARN for a certificate https://kubernetes.io/docs/concepts/services-networking/service/#ssl-support-on-aws
BUT the aws-load-balancer-controller takes a stringList of certificate ARNs allowing you to use multiple certificates.

I had a really hard time figuring out that those aws-load-balancer* annotations are standard k8s but if you install the aws-load-balancer-controller then it replaces how service resources are handled and it almost handles the same annotations in the same way. The important difference for me is the handling of multiple certificates in the service.beta.kubernetes.io/aws-load-balancer-ssl-cert annotation now becomes possible.

@RichardoC
Copy link

RichardoC commented Mar 14, 2022

Hi All,
Do these following health check annotations work with AWS NLB?
service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
For us, its not reflecting changes in console once we apply them.
Or do we have any specific list of annotations that works well with NLB.
--Tarun

Same for me, when will this be fixed? Thanks.

Did anyone find/open a bug for this? I'm encountering the same on EKS 1.19

Found kubernetes/kubernetes#94546 which fixes the health check issues I was encountering

@Ghilteras
Copy link

Ghilteras commented May 11, 2022

I dont think service.beta.kubernetes.io/aws-load-balancer-type supports nlb
it has to be nlb-ip

@haarchri
Copy link

haarchri commented Jun 27, 2022

is private static IPs supportet via annotations ?

@c4m4
Copy link

c4m4 commented Jun 27, 2022

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

For backwards compatibility, controller still supports the nlb-ip as the type annotation. For example, if you specify

service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip

@c4m4
Copy link

c4m4 commented Jun 27, 2022

is private static IPs supportet via annotations ?

To use an static ip, you need to specify this in the k8s service

@haarchri
Copy link

haarchri commented Jun 27, 2022

we have set the following annotations to our ingress-controller to automatically spawn an nlb in front of our eks/ingress-nginx

kubectl describe service ingress-nginx-controller -n ingress-nginx:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-internal: true
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.19.64.10, 10.19.128.10, 10.19.192.10
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb

But the nlb always spawns with dynamic IPs.

@RichardoC
Copy link

RichardoC commented Sep 5, 2022

we have set the following annotations to our ingress-controller to automatically spawn an nlb in front of our eks/ingress-nginx

kubectl describe service ingress-nginx-controller -n ingress-nginx:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-internal: true
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.19.64.10, 10.19.128.10, 10.19.192.10
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb

But the nlb always spawns with dynamic IPs.

Are you using the AWS Load Balancer Controller or what's built in to EKS? That annotation only seems to be used by the AWS LBC which needs to be installed separately.
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/

@kamaraj-muthupandian
Copy link

kamaraj-muthupandian commented Nov 24, 2022

is http2 supported via annotations ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment