Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
AWS ELB-related annotations for Kubernetes Services (as of v1.12.0)

AWS Service annotations

  • service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval (in minutes)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name
  • service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix
  • service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags (comma-separated list of key=value)
  • service.beta.kubernetes.io/aws-load-balancer-backend-protocol (http|https|ssl|tcp)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout (in seconds)
  • service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout (in seconds, default 60)
  • service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-extra-security-groups (comma-separated list)
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
  • service.beta.kubernetes.io/aws-load-balancer-internal (true|false)
  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert (IAM or ACM ARN)
  • service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports (default '*')
  • service.beta.kubernetes.io/aws-load-balancer-type: nlb
@KIVagant

This comment has been minimized.

Copy link

KIVagant commented May 17, 2017

@dod38fr

This comment has been minimized.

Copy link

dod38fr commented May 29, 2017

Thanks for the list.

aws-load-balancer-internal annotation value is only used as a boolean. Why is 0.0.0.0/0 shown as a default value ?

@tuannvm

This comment has been minimized.

Copy link

tuannvm commented Oct 17, 2017

To add additional tags for ELB, reference:

service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags
@srossross-tableau

This comment has been minimized.

Copy link

srossross-tableau commented Nov 15, 2017

Do you know how I can do ssl termination? I e. if service.beta.kubernetes.io/aws-load-balancer-backend-protocol is set to https, then the "Load Balancer Protocol" and the "Instance Protocol" are both set to https. I would like the "Instance Protocol" to remain http.

@tommyo

This comment has been minimized.

Copy link

tommyo commented Nov 16, 2017

@srossross-tableau
service.beta.kubernetes.io/aws-load-balancer-backend-protocol is for the Instance protocol. The 2 ssl annotations are for the load balancer settings. What you want looks something like this:

    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:...."
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "*"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
@downneck

This comment has been minimized.

Copy link

downneck commented Jan 5, 2018

@srossross-tableau @tommyo

service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "*" will terminate ssl on all ports. if your LB is going to be serving any ports without ssl termination (http, ssh, etc.) you want aws-load-balancer-ssl-ports to list only the ports that will terminate ssl (eg. 443)

@c4m4

This comment has been minimized.

Copy link

c4m4 commented Feb 16, 2018

Why there is no tag to say what subnet to use?

@jryberg

This comment has been minimized.

Copy link

jryberg commented Feb 28, 2018

@c4m4, did you figure that one out? I'm also looking to select a specific subnet per load balancer.

@aprisniak

This comment has been minimized.

Copy link

aprisniak commented Mar 5, 2018

How can I redirect to https?

@edify42

This comment has been minimized.

Copy link

edify42 commented Mar 7, 2018

@aprisniak You can with an Ingress definition which can do a HTTP 301/302.

Not sure if there's a nice way to do HTTPS redirects with an ALB type AWS LB

@mt-inside

This comment has been minimized.

Copy link

mt-inside commented Apr 9, 2018

Do you have an updated version of this? It would be useful. E.g. in 1.9+ there's an option to make an NLB.
https://kubernetes.io/docs/concepts/services-networking/service/#network-load-balancer-support-on-aws-alpha

   metadata:
      name: my-service
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
@claylaut

This comment has been minimized.

Copy link

claylaut commented May 22, 2018

How can I enable sticky sessions?

@dorsegal

This comment has been minimized.

Copy link

dorsegal commented May 24, 2018

Do you know how to add extra security groups?

@seh

This comment has been minimized.

Copy link

seh commented May 29, 2018

NB: At present, service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled is not honored for NLBs.

@codrinbucur

This comment has been minimized.

Copy link

codrinbucur commented Aug 23, 2018

Is there an option to automatically deploy the LB in multiple availability zones? Or could the "aws-load-balancer-cross-zone-load-balancing-enabled=true" create the additional availability zones' instances?

@ksemaev

This comment has been minimized.

Copy link

ksemaev commented Aug 29, 2018

@mt-inside have you found any updated info?

@cdenneen

This comment has been minimized.

Copy link

cdenneen commented Aug 30, 2018

@mgoodness Agree with @dod38fr the 0.0.0.0/0 annotation shouldn't be used anymore as it's very confusing.
Also @dod38fr while it's a boolean you can't use a boolean if you try true it fails... must be 'true'

@tomweston

This comment has been minimized.

Copy link

tomweston commented Dec 17, 2018

Has anyone had any luck using the following annotations on AWS Classic ELB’s:

service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix

No matter what values I give them, they don’t seem to register, forget the ELB to go into a pending state and never deploy.
Attaching certs works fine though 🙂

service.beta.kubernetes.io/aws-load-balancer-ssl-cert
@benejo

This comment has been minimized.

Copy link

benejo commented Jan 25, 2019

@tomweston
Try adding the bucket policies as mentioned here before applying

service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name

annotation.

@kesor

This comment has been minimized.

Copy link

kesor commented Jan 31, 2019

@codrinbucur @jryberg @c4m4 you need to tag the subnet with kubernetes.io/cluster/name_of_cluster: shared and it will be picked up by the LB creation process and add it as a subnet.

@tprakash17

This comment has been minimized.

Copy link

tprakash17 commented Apr 12, 2019

Hi All,

Do these following health check annotations work with AWS NLB?

service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold

For us, its not reflecting changes in console once we apply them.

Or do we have any specific list of annotations that works well with NLB.

--Tarun

@dmildh

This comment has been minimized.

Copy link

dmildh commented May 1, 2019

Hello everyone. Has anyone tried sending the elb logs to a bucket in another account? The bucket policy is setup so that I can manually add it to the elb but when trying to use the following it does not add it. The same options can create a bucket within the same account.

service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval="5"
service.beta.kubernetes.io/aws-load-balancer-access-log-enabled="true"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name="central-bucket-name"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix="my-useful-prefix"

@manojchandrabss

This comment has been minimized.

Copy link

manojchandrabss commented May 7, 2019

Hi @everyone

is there any annotation we have in service alb for target-type : ip internal facing. we are stucked in prod.

@fgreg

This comment has been minimized.

@stowns

This comment has been minimized.

Copy link

stowns commented Sep 2, 2019

it's kind of insane that I can't find these anywhere in AWS's actual documentation...am I missing something?

@mgoodness

This comment has been minimized.

Copy link
Owner Author

mgoodness commented Sep 2, 2019

@gaffneyd4

This comment has been minimized.

Copy link

gaffneyd4 commented Sep 6, 2019

@dod38fr

Thanks for the list.

aws-load-balancer-internal annotation value is only used as a boolean. Why is 0.0.0.0/0 shown as a default value ?

@cdenneen

@mgoodness Agree with @dod38fr the 0.0.0.0/0 annotation shouldn't be used anymore as it's very confusing.
Also @dod38fr while it's a boolean you can't use a boolean if you try true it fails... must be 'true'

It looks like the code just checks for if the annotation is present and is set to anything other than an empty string "".
https://github.com/kubernetes/kubernetes/blob/e4b0a935fa393944b6322fa6ef0970d858ad70f6/pkg/cloudprovider/providers/aws/aws.go#L3335

That would explain why anything goes, 0.0.0.0/0 or "true"

@dhananjaya-senanayake

This comment has been minimized.

Copy link

dhananjaya-senanayake commented Sep 27, 2019

@tprakash17 did you find any specific list of annotations that works well with NLB

@nikosheng

This comment has been minimized.

Copy link

nikosheng commented Oct 29, 2019

@tprakash17 did you find any specific list of annotations that works well with NLB

me too.

@tehale

This comment has been minimized.

Copy link

tehale commented Nov 6, 2019

With nlb these annotations are not working?

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:***"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http

@Firefishy

This comment has been minimized.

Copy link

Firefishy commented Nov 7, 2019

With nlb these annotations are not working?

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:**"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "
"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http

NLB and SSL cert require Kubernetes 1.15. See kubernetes/kubernetes#74910 @tehale

@mike503

This comment has been minimized.

Copy link

mike503 commented Nov 19, 2019

I'm trying to use some of these annotations with EKS.

# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-eks-e9b1d0", GitCommit:"e9b1d0551216e1e8ace5ee4ca50161df34325ec2", GitTreeState:"clean", BuildDate:"2019-09-21T08:33:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

I've commented out the ones that seem to mess things up. With the 3 that are uncommented, it works as expected. If I uncomment and try one of the ones commented, for example aws-load-balancer-cross-zone-load-balancing-enabled, it winds up ignoring ALL annotations, so the SSL certificate is ignored, everything is ignored and it's like none of the annotations exist. I don't see any errors that come out anywhere, however. Am I missing something? AFAIK these are supported under the kubectl and server versions.

  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:REDACTED:certificate/REDACTED
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "*"
# these are supposed to work, but seem to break things.
# ref: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1
#    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
#    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: 2
#    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: 5
#    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: 3
#    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: 2

Also just a wishlist note - I've seen some comments, maybe even a patch, but supporting naming the load balancer so it's not the randomized hostname, but actually something that can be requested and friendly. :)

@marcellodesales

This comment has been minimized.

Copy link

marcellodesales commented Nov 28, 2019

For all the boolean values, make sure to use the values as string to avoid getting errors like kubernetes/kubernetes#59113

@boddumanohar

This comment has been minimized.

Copy link

boddumanohar commented Nov 29, 2019

when creating a k8s Service of type load balancer on AWS, under annotations sections, I specify

service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

who intercepts this annotation and how will k8s get permission to provision an aws load balancer?

@M00nF1sh

This comment has been minimized.

Copy link

M00nF1sh commented Dec 5, 2019

@mike503, all boolean and numbers needs to be quoted, otherwise, it will cause other annotations to be lost silently as well when u do kubectl apply. You can verify that by doing kubectl get service <your-service> -o yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.